aboutsummaryrefslogtreecommitdiffstats
path: root/includes/libs/objectcache/BagOStuff.php
Commit message (Collapse)AuthorAgeFilesLines
...
* Fix typos in comments (A-B)Siddharth VP2021-12-261-1/+1
| | | | Change-Id: I852453fbeeebdc4e34c0b35c0fdca4b4ab74fde9
* Remove or replace usages of "sanity"Reedy2021-11-191-1/+1
| | | | | Bug: T254646 Change-Id: I2b120f0b9c9e1dc1a6c216bfefa3f2463efe1001
* objectcache: add watchErrors() to BagOStuff/WANObjectCacheAaron Schulz2021-11-101-10/+70
| | | | | | | | | | | | | | | | | | | The new style of checking for the last error during a section of calls is more robust since it allows nesting of callers. Typically, an external caller will want to watch a section of code that will involve zero or more internally watched sections. Errors that are seen internally (leading to a failing response) should also be visible externally. Replace internal BagOStuff clearLastError() calls. Replace WANObjectCache clearLastError() calls. Such a class should not clear the error codes since the class is effectively "internal". Callers that are more meaningfully "external" might want to check the errors. Cleanup "last" error handling for proxy backends. Change-Id: I281817a85602967c0ec2bdd23a5d8be101680b64
* objectcache: remove deprecated setDebug() methodAaron Schulz2021-10-131-14/+0
| | | | Change-Id: I4702d7c8df2c0a645279ed59b6b7a9aa9e5cea34
* objectcache: make use of new `modtoken` field in SqlBagOStuff (ii)Aaron Schulz2021-08-201-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a multi-primary mode option that supports MySQL DB setups that use circular replication with STATEMENT formatted binlogs. The `modtoken` column is only used when multi-primary mode is explicitly enabled in configuration. The column is used by write queries to determine the "winning" version of keys, with the goal of approximating "Last-Write-Wins" eventual consistency. Writes with different timestamps can be handled by picking the one with the highest timestamp as the "winner". Writes with the same timestamp, from different primary DBs, can be handled by picking the one from the primary DB with the highest server_id. Writes with the same token timestamp from the same primary DB can be handled by picking the last write to appear in the binlog. The delete() operation uses tombstones in multi-primary mode, since there must be a key version to actually compare with the versions from other operations. Also: * Remove "LOCK IN SHARE MODE" that was made obsolete by the CONN_TRX_AUTOCOMMIT flag. For the SQLite transaction case, it is serializable anyway. * Simplified handleWriteError() to match handleReadError() and merged them into handleDBError(). Changes from d56a686f83eb7d19a: * Keep using replace() for set() in non-multi-primary mode. * Fix old encoding issues when incrementing keys with postgres. Bug: T274174 Change-Id: I25b1278586f99fdace8f2a081f6f460f734e5d83
* Merge "Revert "objectcache: make use of new `modtoken` field in SqlBagOStuff""jenkins-bot2021-08-171-1/+1
|\
| * Revert "objectcache: make use of new `modtoken` field in SqlBagOStuff"Ladsgroup2021-08-171-1/+1
| | | | | | | | | | | | | | | | | | This reverts commit d56a686f83eb7d19abe9aced0af1985f25025901. Reason for revert: T288998#7287717 Bug: T288998 Change-Id: Ia05b00a345335fe8bdd6041323c3abfa79ded169
* | Merge "objectcache: BagOStuff::genericKeyFromComponents() empty component fix"jenkins-bot2021-08-121-2/+2
|\ \ | |/ |/|
| * objectcache: BagOStuff::genericKeyFromComponents() empty component fixAaron Schulz2021-08-111-2/+2
| | | | | | | | | | | | | | | | Empty leading components should still be colon separated. Although the keyspace and collection should never be empty strings, it does not hurt to handle that case anyway for safety. Change-Id: Ic89acaf1fe5c34e58323b2913a9a26f973868590
* | objectcache: make use of new `modtoken` field in SqlBagOStuffAaron Schulz2021-08-051-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a multi-primary mode option that supports MySQL DB setups that use circular replication with STATEMENT formatted binlogs. The `modtoken` column is only used when multi-primary mode is explicitly enabled in configuration. The column is used by write queries to determine the "winning" version of keys, with the goal of approximating "Last-Write-Wins" eventual consistency. Writes with different timestamps can be handled by picking the one with the highest timestamp as the "winner". Writes with the same timestamp, from different primary DBs, can be handled by picking the one from the primary DB with the highest server_id. Writes with the same token timestamp from the same primary DB can be handled by picking the last write to appear in the binlog. The delete() operation uses tombstones in multi-primary mode, since there must be a key version to actually compare with the versions from other operations. Also: * Remove "LOCK IN SHARE MODE" that was made obsolete by the CONN_TRX_AUTOCOMMIT flag. For the SQLite transaction case, it is serializable anyway. * Simplified handleWriteError() to match handleReadError() and merged them into handleDBError(). Bug: T274174 Change-Id: Icc5eff9a032dd3403b5718058f20e38f8ea84af5
* | build: Updating dependencieslibraryupgrader2021-07-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | composer: * mediawiki/mediawiki-codesniffer: 36.0.0 → 37.0.0 The following sniffs now pass and were enabled: * Generic.ControlStructures.InlineControlStructure * MediaWiki.PHPUnit.AssertCount.NotUsed npm: * svgo: 2.3.0 → 2.3.1 * https://npmjs.com/advisories/1754 (CVE-2021-33587) Change-Id: I2a9bbee2fecbf7259876d335f565ece4b3622426
* | objectcache: make BagOStuff::lock/unlock easier to overrideAaron Schulz2021-07-191-18/+8
|/ | | | | | | | | | | | | | | Avoid the duplication of the reentrant lock logic in subclasses. Move "lock already expired" logic from getScopedLock() to unlock() so that it applies more generally. Warn when unlock() is called on a key that is not even locked as well. Also: * Rename lock() $expiry argument to $exptime for consistency * Fix return types for serialize()/unserialize() Bug: T274174 Change-Id: I211536e616cf7f1cc60181c378bbf9b35ffa40a4
* purgeParserCache.php: Implement --tag for purging one server onlyTimo Tijhof2021-06-241-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Large wiki farms may have to purge servers concurrently (instead of one at a time) in order to keep up with new writes and delete expired rows faster than new rows are written. The parameter for this uses server tags for three reasons: * Maintenance risk and complexity. This requires the least information about MW configuration to be hardcoded in the scheduled maintenance cronjob (compared to: server indexes which are a runtime concept, or specific hostnames/IP/ tableprefixes which may change and should not require coordinating changes elsewhere). * Operational convenience. By using server tags, the parameters don't have to vary between data centers. * Code complexity. The current code for obtaining connections is based on server indexes, which are easy to mapped at runtime from server tags. Other ways of identifying shard like hostnames are either an awkward fit (as they don't uniquely identify a shard per-se, with multiple instances on the same hardware at WMF), or require SqlBagOStuff to store and maintain more information about connections than it currently has readily available. Bug: T282761 Change-Id: I618bc1e8ca3008a4dd54778ac24aa5948f27c52e
* build: Swap deprecated @codingStandardsIgnore to phpcs:ignoreUmherirrender2021-04-041-2/+0
| | | | | Bug: T278594 Change-Id: I09a6175917090593e6e0055203a890c32bea03a5
* Add $wgChronologyProtectorStash and improve $wgMainStash commentsAaron Schulz2021-03-011-9/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | Remove WRITE_SYNC flag from ChronologyProtector since the current plan is to simply use a datacenter-local storage cluster. Move the touched timestamps into the same stash key that holds the replication positions. Update the ChronologyProtector::getTouched() comments. Also: * Use $wgMainCacheType as a $wgChronologyProtectorStash fallback since the main stash will be 'db-replicated' for most sites. * Remove HashBagOStuff default for position store since that can result in timeouts waiting on a write position index to appear since the data does not actually persist accress requests. * Rename ChronologyProtector::saveSessionReplicationPosition() since it does not actually save replication positions to storage. * Make ChronologyProtector::getTouched() check the "enabled" field. * Allow mocking the current time in ChronologyProtector. * Mark some internal methods with @internal. * Migrate various comments from $wgMainStash to BagOStuff. * Update some other ObjectCache related comments. Bug: T254634 Change-Id: I0456f5d40a558122a1b50baf4ab400c5cf0b623d
* objectcache: Always enable debug mode in BagOStuffTim Starling2021-01-291-3/+7
| | | | | | | | | | | | BagOStuff has a debug mode because MemcachedClient has a debug mode, because Ryan T. Dean put one in there in 2003. It's not strictly necessary now that we have log levels. This means that a few existing $this->logger->debug() calls are now on the same effective level as $this->debug() calls, they can't be enabled separately. But that seems inconsequential. Change-Id: I4d1cb9153559b29468b6ca530bbc56e2b35d1721
* objectcache: add statsd key metrics to BagOStuff classesAaron Schulz2021-01-251-9/+86
| | | | | | | | | Update SQL, REST, and redis subclasses to emit call count and payload size metrics for cache key operations. These metrics are bucketed by cache key collection (similar to WANCache). Bug: T235705 Change-Id: Icaa3fa1ae9c8b0f664c26ce70b7e1c4fc5f92767
* objectcache: make BagOStuff key encoding more consistentAaron Schulz2020-12-021-23/+165
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add "generic" key methods for quickly deriving keys from key component lists in a bijective manor. This is useful for BagOStuff classes that wrap other BagOStuff instances or for parsing keys to get stats. Make the proxy BagOStuff classes (ReplicatedBagOStuff, MultiWriteBagOStuff, CachedBagOStuff) use "generic" keys so that they can convert to appropriate keys when making backing cache instance method calls. Make EmptyBagOStuff, HashBagOStuff, APCUBagOStuff, RedisBagOStuff, and RESTBagOStuff use "generic" keys rather than those of MediumSpecificBagOStuff::makeKeyInternal(). This lets proxy BagOStuff classes bypass key conversions when used with instances of these classes as backing stores. Also: * Fix missing incr(), incrWithInit(), and decr() return values in MultiWriteBagOStuff. * Make MultiWriteBagOfStuff, ReplicatedBagOStuff, and CachedBagOStuff use similar backend method forwarding styles by using a new BagOStuff method. * Improved various related bits of documentation. Bug: T250239 Bug: T235705 Change-Id: I1eb897c2cea3f5b756dd1e3c457b7cbd817599f5
* objectcache: rename BagOStuff::makeKeyInternal() arguments and update ↵Aaron Schulz2020-11-231-2/+3
| | | | | | documentation Change-Id: I9e7119d7822eae945f0d627d1abcf491989aa882
* Replace "@stable for subclassing" with "@stable to extend"daniel2020-07-131-1/+1
| | | | | | | | | | | For compliance with the new version of the table interface policy (T255803). This patch was created by an automated search & replace operation on the includes/ directory. Bug: T257789 Change-Id: Ie32c1b11b3d16ddfc0c83a757327d449ff80b2e4
* Annotate extensible classes.daniel2020-07-091-0/+1
| | | | | | | This marks some base classes as stable for subclassing. Bug: T247862 Change-Id: I7b77075849aa40960f2486b463e498a3ea8058e9
* Fix more libs PSR12.Properties.ConstantVisibility.NotFoundReedy2020-05-161-7/+7
| | | | Change-Id: If3bac6b0ff6fbb89bfa2b9fa91809135a76c610b
* objectcache: improve documentation for BagOStuff::setNewPreparedValues()Aaron Schulz2020-04-181-6/+15
| | | | Change-Id: I0a30da88e693a3a0ad43b70ab5724e361a07149f
* objectcache: add BagOStuff::setNewPreparedValues()Aaron Schulz2020-03-261-0/+18
| | | | | | | | This can be used by WANObjectCache or statsd metrics to estimate the size of objects and network commands without having to do an extra serialization round. Change-Id: I7b1b787bba39a21e2e04262e77ee7ae24e88f8ca
* objectcache: Inject Config object to ObjectCache::newFromParamsTimo Tijhof2020-03-041-0/+8
| | | | | | | | | | | | | | | | | | | | * Avoid direct $GLOBALS lookups. * Avoid MediaWikiServices singleton for Config object. Also given that newFromParams() constructs its Logger object unconditionally as of I2aa835c5ec79, stop creating it ahead of time in ServiceWiring. This code, including the default loggroup value, was duplicated by me from ObjectCache.php to ServiceWiring.php in commits 3828558140e and bd16c5eb34e9a5 because I needed to obtain the Logger object in order to send a message to it about how it was created. Solve this debt by letting ServiceWiring access the actual logger that was created by newFromParams() through a new getLogger() object. Change-Id: Ib2de7b22f6c79db0c700fae06269d04fbe27831d
* objectcache: split out StorageAwareness/ExpirationAwareness from IExpiringStoreAaron Schulz2020-03-021-1/+8
| | | | | | | Add more detailed QOS_* constants while at it Bug: T236412 Change-Id: Ia862c5111a3daf10a34fc78163301629228efa6b
* objectcache: rename $ttl => $exptime in BagOStuff for consistencyAaron Schulz2020-01-091-8/+8
| | | | Change-Id: Ie09176eb591faa525169de3dc1e861b9610d90e5
* Document more scalar types for BagOStuff::makeKey and Language::formatNumDaimona Eaytoy2019-12-151-2/+2
| | | | | | | | | | | | | | These were spotted by phan's stricter type checks. All make*Key methods can also take integers, and that's pretty common across core and extensions. Language::formatNum can also take numbers (and tests cover this case); in this case, move the cast to the top of the method, so that we're passing a string to strtr, too. The methods touched in this patch are the main source of false positives in both core and extensions. Change-Id: I6d538ce3ca7fd2d495c2bafbab7cc279da69db1c
* Unsuppress phan issues part 6Daimona Eaytoy2019-09-011-0/+1
| | | | | | Bug: T231636 Depends-On: I50377746f01749b058c39fd8229f9d566224cc43 Change-Id: I2cd24e73726394e3200a570c45d5e86b6849bfa9
* objectcache: improve BagOStuff arithmetic method signaturesAaron Schulz2019-08-241-9/+14
| | | | | | | | | | | Make the default $init value for incrWithInit() be $value. This is far less suprising and also makes the operation easier to replicate without conflicts. Make decr() definitions more explicit since various cache drivers do not handle negative incr() values (e.g. memcached). Change-Id: I2b8d642656cc91c841abbd7a55d97eba101b027a
* objectcache: add BagOStuff::fieldHasFlags() convenience methodAaron Schulz2019-08-221-0/+10
| | | | Change-Id: Id2bf4ec0dd0999f988b70a895003c4b1aaae51a2
* objectcache: fix failing tests for non-HashBagOStuff backendsAaron Schulz2019-08-071-1/+1
| | | | | | | | | | | | | | | Use real time for testing absolute expirations with changeTTL(). Otherwise, backends like memcached or redis will fail since they do not use the mock time. Also: * Make SqlBagOStuff actually override changeTTLMulti() by using the right method name * Check TTL_INDEFINITE more explicitly for clarity * Rename TTL conversion methods for clarity * Use isRelativeExpiration() in MemcachedBagOStuff Change-Id: I9365ceb31d4e7bef65906363d42b8c3020a66346
* Merge "objectcache: let BagOStuff::getWithSetCallback() callbacks modify the ↵jenkins-bot2019-07-251-5/+3
|\ | | | | | | TTL"
| * objectcache: let BagOStuff::getWithSetCallback() callbacks modify the TTLAaron Schulz2019-07-231-5/+3
| | | | | | | | | | | | | | Also simplify the code by removing the is_callable() check and relying on regular PHP errors instead of an exception for bad callbacks. Change-Id: I084b0132c5fb05f1941a6d6839cfa74e2cf677f0
* | objectcache: Use variadic signature for makeKey()Timo Tijhof2019-07-231-6/+6
|/ | | | | | | | | | | | | | This should help fix the following issues that various repos are getting from Phan as of late: > Call with 5 arg(s) to \BagOStuff::makeKey() which only takes 2 arg(s) > defined at ../../includes/libs/objectcache/BagOStuff.php:456 > <source="PhanParamTooMany"/> Bug: T228563 Depends-On: I5cfba063821101325a5a7359e6b8ad71a0fb1b2f Depends-On: Ifa5b96735376f2fbe3680799f960616ba8d357ff Change-Id: Ic9df7f3ad7f356c7cbdfe1edfbe35821b931dda6
* objectcache: add MediumSpecificBagOStuff base class for non-proxy subclassesAaron Schulz2019-07-191-648/+38
| | | | | | This make it much clearer what needs to be overridden Change-Id: I3073f8a0605f557c6a3a93d0d8401cddd0fb8dbe
* Merge "objectcache: optimize MemcachedPeclBagOStuff::*Multi() write methods"jenkins-bot2019-07-181-8/+13
|\
| * objectcache: optimize MemcachedPeclBagOStuff::*Multi() write methodsAaron Schulz2019-07-151-8/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Make use of Memcached::OPT_NO_BLOCK and Memcached::OPT_BUFFER_WRITES at selective points to avoid slow internal blocking for loops in the PECL extension methods. As this mode makes the return values less meaningful, this is only triggered only if WRITE_BACKGROUND is provided (it only catches some glaring format or size errors that the client runs into). Leave changeTTLMulti() to the default as the above approach would not make it faster, since the diver does not buffer for that command. Expand mctest.php to include *Multi() methods and also break down the single key method timings by method. Change-Id: I11fc58e1a2296778b2526a0b2f3103fef1baba0c
* | objectcache: make BagOStuff::getMulti() preserve order and omit keys with ↵Aaron Schulz2019-07-141-7/+15
| | | | | | | | | | | | | | | | | | missing segments A key missing segments should count as non-existing, meaning that it should not appear in the return value at all Change-Id: I0f08b3dba77e0d335ebb2d75061927ebd36570d2
* | objectcache: normalize BagOStuff method overriding pattern for *Multi() methodsAaron Schulz2019-07-111-32/+17
|/ | | | Change-Id: I1bebb60307b1a166461cb5f9a55a79194cc0e363
* bagostuff: optimize SqlBagOStuff and fix failing segmentation testsAaron Schulz2019-07-091-11/+73
| | | | | | | | | | | | | | | | | | | | | In SqlBagOStuff: * Add modifyMulti() helper method to reduce code duplication * Improve atomicity of add(), cas(), and changeTTL() queries * Avoid integer serialization and improve atomicity of incr() * Optimize new BagOStuff::changeTTLMulti() method In BagOStuff: * Add changeTTLMulti() method for subclasses to optimize * Make set() ignore WRITE_ALLOW_SEGMENTS for integers so incr() works * Strip WRITE_ALLOW_SEGMENTS flag from the setMulti() call in set() to avoid triggering bogus sanity check exceptions * Fix BagOStuffTest::testSetSegmentable failures via the above changes * Enforce WRITE_ALLOW_SEGMENTS sanity check in setMulti() for all the subclasses by using a final wrapper method * Add WRITE_ALLOW_SEGMENTS sanity check to deleteMulti() Bug: T113916 Change-Id: I25d1790fa9b0d1837643efccfa94a12043cfbf42
* objectcache: deleteObjectsExpiringBefore() signature and code improvementsAaron Schulz2019-07-091-3/+7
| | | | | | | | | | Make the method use a callable type hint for $progressCallback. Also make the SQLBagOStuff subclass shuffle() the array of server and table indexes in case the limit keeps getting applied. Only garbage collect on DBs that were going to be written to anyway. Also use DeferredUpdates if possible. Change-Id: I723e6377c26750ff98e33f7ab103c6162ae65f43
* Merge "objectcache: tweak SqlBagOStuff purging to happen on write only"jenkins-bot2019-07-081-2/+3
|\
| * objectcache: tweak SqlBagOStuff purging to happen on write onlyAaron Schulz2019-07-061-2/+3
| | | | | | | | | | | | | | | | Also adjust the frequency default and put in a 100 row limit for randomized purges. This adds a $limit parameter to the BagOStuff method deleteObjectsExpiringBefore(). Change-Id: I66bfcc67e8e118f8c659dbbee13d54bf2cd937f5
* | objectcache: add IStoreKeyEncoder interface for key generation helper methodsAaron Schulz2019-07-081-1/+1
|/ | | | | | | If a key is stored in both BagOStuff and WANObjectCache, it useful to have one common method for generating the cache and storage keys. Change-Id: I8b77651746a55e8a98a79db298226e334d15a494
* objectcache: optimize lock() and unlock() methods in SqlBagOStuffAaron Schulz2019-07-041-1/+5
| | | | | | | Also clean up base method in BagOStuff. Bug: T113916 Change-Id: I3a1c6afe531a8eae34608bc7fe0aa6f9f4d439fe
* Merge "objectcache: add object segmentation support to BagOStuff"jenkins-bot2019-06-131-28/+265
|\
| * objectcache: add object segmentation support to BagOStuffAaron Schulz2019-06-111-28/+265
| | | | | | | | | | | | | | | | | | | | | | | | Use it for ApiStashEdit so that large PaserOutput can be stored. Add flag to allow for value segmentation on set() in BagOStuff. Also add a flag for immediate deletion of segments on delete(). BagOStuff now has base serialize()/unserialize() methods. Bug: T204742 Change-Id: I0667a02612526d8ddfd91d5de48b6faa78bd1ab5
* | objectcache: Make BagOStuff::getCurrentTime() publicLucas Werkmeister2019-06-091-2/+4
|/ | | | | | | | If another cache wraps a BagOStuff, it may need to know about the BagOStuff’s notion of the current time, which may be mocked and differ from the real time. Change-Id: I40f4085d4cf549314394d3140cccb1fe4e48690a
* objectcache: remove some unneeded code from BagOStuff::mergeViaCasAaron Schulz2019-03-271-5/+2
| | | | Change-Id: Ib92090fe4e1b1eb2c4f9e031f8476b24189392c8
generated by cgit v1.2.3 (git 2.25.1) at 2025-04-22 22:58:15 +0000