aboutsummaryrefslogtreecommitdiffstats
path: root/maintenance/storage
Commit message (Collapse)AuthorAgeFilesLines
* maintenance: Use type declaration on undocumented private functionsUmherirrender2025-03-314-10/+10
| | | | Change-Id: I2c1bc3b4e0ee782076ddaeae133672b730d345d9
* Replace a number of uses of class aliases with the real thingsJames D. Forrester2025-03-269-0/+11
| | | | | | | Found via disabling phan's alias support for a run (and ignoring wgLang's hard-coded state of being a \Language, alas). Change-Id: I4753bcd84d72d6de111fc7ffc5841fa417bf7333
* Remove trivial 1-line PHPDocs that just repeat the codethiemowmde2025-01-161-3/+0
| | | | | | | | | | | | | I assume these are all either auto-generated by an IDE or the language-level type declarations have been added later. In any case the comments don't add any new information to what the code already says. This is just extra clutter that makes the code harder to read, I would argue. There are many, many more comments like this. In this patch I intentionally focus on the most trivial 1-line comments. Change-Id: Ia294bf4ce0d8a77036842fe25884bc175c2b0e7d
* build: Upgrade mediawiki/mediawiki-phan-config from 0.14.0 to 0.15.0James D. Forrester2024-12-191-0/+1
| | | | | | | | | | | | | Suppress remaining issues Depends-On: I633ec05fe9ab6a815225911d942ecb79f21c428f Depends-On: I186e799256fbaf5ee77558bd146f9418dd5eaacc Depends-On: Ib84b7d71e8308a36409f30ecfd16e9de149e97b3 Depends-On: I26aa4adf31b4142248abe3e0f9aa8314512602a6 Depends-On: Icf24e208a487bafe3d1983536870aac19cfc4b5e Depends-On: I99915b893f14cfbb2b54d9a6264b4f294200c4f2 Depends-On: Ia53b71dff3299837856ae3c004cd0227bd45e05b Change-Id: I825dce05d4a6ca87acf25063e3664cb2d16f5db0
* externalstore: cleanup ExternalStoreDB::getTable() and fix callersAaron Schulz2024-09-182-3/+5
| | | | | | | | | | | Remove unused $db parameter and make the $cluster required. Update the checkStorage.php callers to provide the cluster. Use the getTable() method in trackBlobs.php to avoid duplication. Mark this method as @internal to the class and /storage scripts. Change-Id: I888040a536a60e22e780900a59e4c34b6c468cdf
* maintenance: Add missing documentation to class propertiesUmherirrender2024-09-133-0/+26
| | | | | | | | | | Add doc-typehints to class properties found by the PropertyDocumentation sniff to improve the documentation. Once the sniff is enabled it avoids that new code is missing type declarations. This is focused on documentation and does not change code. Change-Id: I7dec01892a987a87b1b79374a1c28f97d055e8fa
* Exclude boilerplate maintenance code from code coverage reportsDreamy Jazz2024-08-2710-0/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | Why: * Maintenance scripts in core have bolierplate code that is added before and after the class to allow directly running the maintenance script. * Running the maintenance script directly has been deprecated since 1.40, so this boilerplate code is only to support a now deprecated method of running maintenance scripts. * This code cannot also be marked as covered, due to PHPUnit not recognising code coverage for files. * Therefore, it is best to ignore this boilerplate code in code coverage reports as it cannot be marked as covered and also is for deprecated code. What: * Wrap the boilerplate code (requiring Maintenance.php and then later defining the maintenance script class and running if the maintenance script was called directly) with @codeCoverageIgnore comments. * Some files use a different boilerplate code, however, these should also be marked as ignored for coverage for the same reason that coverage is not properly reported for files. Bug: T371167 Change-Id: I32f5c6362dfb354149a48ce9c28da9a7fc494f7c
* Make use of the ??= and ?? operators where it makes sensethiemowmde2024-08-261-7/+4
| | | | | | | | This touches various production classes and maintenance scripts. The code should do the exact same as before. The main benefit is that the syntax avoids any repetition. Change-Id: I5c552125469f4d7fb5b0fe494d198951b05eb35f
* maintenance: remove Database::clearFlag() call in recompressTracked.phpAaron Schulz2024-08-081-1/+0
| | | | | | | Maintenance scripts do not use DBO_TRX mode so this was redundant. Bug: T311090 Change-Id: I6dc6d2a8e7daf8a2a1dac7c3f8968b4a575b3dd6
* maintenance: Use expression builder instead of raw sqlUmherirrender2024-07-224-7/+7
| | | | | Bug: T361023 Change-Id: Ieb229d8088cb1ff3f03e44f7ac99eb612f48bc7b
* rdbms: Create IReadableDatabase::andExpr() / ::orExpr()Umherirrender2024-07-111-2/+1
| | | | | | | | | | | | | | | | | | | | | Avoid the call to internal constructor of AndExpressionGroup and OrExpressionGroup by creating a factory function similiar as the IReadableDatabase::expr function for Expression objects. This is also a replacement for calls to ISQLPlatform::makeList with LIST_AND or LIST_OR argument to reduce passing sql as string to the query builders. Created two functions to allow the return type to be set for both expression group to allow further calls of ->and() or ->or() on the returned object. Depending on the length of the array argument to makeList() it is sometimes hard to see if the list gets converted to AND or OR, having the operator in the function name makes it easier to read, so two functions are helpful in this case as well. Bug: T358961 Change-Id: Ica29689cbd0b111b099bb09b20845f85ae4c3376
* Use namespaced classes (1)Umherirrender2024-06-161-0/+1
| | | | | | | Changes to the use statements done automatically via script Addition of missing use statement done manually Change-Id: Ic4d4dd61de5ab896fb6173eb579c81f164a1e4a3
* rdbms: Remove IReadableDatabase::getReplicaPos()Amir Sarabadani2024-04-302-8/+2
| | | | | | | Completely unused. Bug: T363839 Change-Id: I041ab5ce57ef116076dcc07b2035b5336ceff032
* Stop using LoadBalancer::getConnectionRef() so it can be hard-deprecatedAmir Sarabadani2024-04-301-8/+1
| | | | | Bug: T326274 Change-Id: I90493d7cd4c21fdc022bcc19765fc04d986a9c8f
* Merge "Use expression builder to avoid raw sql via BETWEEN operator"jenkins-bot2024-04-233-6/+18
|\
| * Use expression builder to avoid raw sql via BETWEEN operatorUmherirrender2024-04-213-6/+18
| | | | | | | | | | | | Replace BETWEEN with >= and <= operator Change-Id: Ic21b6f4cc11c773c967d9d4c5f20e762c2ff9629
* | maintenance: storage: Fix multiple property declaration phpcs errorsTaavi Väänänen2024-04-213-5/+12
|/ | | | Change-Id: I74302c9b2818e02d7d5e67a728558ced5e8f0181
* maintenance: Migrate to IDatabase::newUpdateQueryBuilderUmherirrender2024-04-143-49/+65
| | | | | Bug: T353219 Change-Id: Ic278c8534dad40a3f34674db2d5fbfbca5984da8
* Replace last remaining wfGetDB() calls in core, except ResourceLoaderJames D. Forrester2024-02-141-10/+14
| | | | | Bug: T330641 Change-Id: I6d30af6ff9f667e367d39befb80c2bb0bf5fb29e
* maintenance: Replace unnecessary uses of LBFactory and LoadBalancerBartosz Dziewoński2024-01-233-10/+5
| | | | | | | | | | * Change `$services->getDBLoadBalancerFactory()->waitForReplication()` to `$this->waitForReplication()` * Change various complicated expressions to `$this->getReplicaDB()` and `$this->getPrimaryDB()` * Remove unused variables Change-Id: Ia857be54938a32bb6288dcdf695a35cd38761c3c
* maintenance: Introduce getReplicaDB() and getPrimaryDB()Amir Sarabadani2024-01-188-28/+28
| | | | | | | | | | | | And start using them instead of wfGetDB(), LB/LBF connection methods or worse, $this->getDB(). $this->getDB() reuses the database object regardless of whether you're calling a replica or primary, leading to returning a replica on a primary and other way around. Bug: T330641 Change-Id: I9e2cf85ca277022284fc26b9f37db57bd12aaa81
* maintenance: Migrate to expression buildersAmir Sarabadani2024-01-174-10/+18
| | | | | | | This was somehow left out Bug: T210206 Change-Id: I70851b5b99fa865dbfd629caf2c1866c85418350
* Migrate remaining Database::insert calls to InsertQueryBuilderAmir Sarabadani2023-12-221-4/+16
| | | | | | | | | Tests are not checked. There is nothing left as far I can check. Bug: T353219 Change-Id: I1d58397118c7ab1110b9d7cf400c59c4bff7378c
* Use thousands separators in selected integer literalsTim Starling2023-12-122-2/+2
| | | | | | | | | | For readability. Allowed since PHP 7.4. I searched for integer literals of 6 or more digits, and also changed some nearby smaller numbers for consistency. Bug: T353205 Change-Id: I8518e04889ba8fd52e0f9476a74f8e3e1454b678
* WikiImporter: Pass Authority for permissions instead of global contextBartosz Dziewoński2023-12-111-1/+3
| | | | | | | | | | | | | | | Pass Authority to WikiImporter constructor, instead of looking at the user from RequestContext::getMain(), and skipping this check if $wgCommandLineMode is true. Maintenance scripts now use UltimateAuthority, to match the original intent of skipping permission checks, see 2ed55f42 / r96311. The Authority parameter to WikiImporterFactory::getWikiImporter() is optional for now for backwards-compatibility. It should become required later after deprecation. Change-Id: Iea1d03dcdcbda2f9a9adbff1b0d319efd22c4d86
* maintenance: Migrate $db->buildLike() to expression builderAmir Sarabadani2023-11-061-8/+21
| | | | | Bug: T210206 Change-Id: Ie7bf3701fa9d51a43167ce7ec0c1f30bc090296b
* maintenance: Return false rather than silently continue on corrupted legacy ↵James D. Forrester2023-10-132-7/+52
| | | | | | | blobs Bug: T340174 Change-Id: I5d6385b5c924985f47e199dee3ecef13905d6388
* Convert numerous DB queries to use QueryBuildersReedy2023-10-092-41/+34
| | | | | Bug: T344971 Change-Id: Ia727b513a6bfcaa5a0b13977a6789aa879ad2f0b
* maintenance: Migrate another batch to SelectQueryBuilderAmir Sarabadani2023-09-214-34/+33
| | | | | | | Around fifty-ish. Found becuase of fixed MigrateSelect. Bug: T344971 Change-Id: If85428d5a033822bfd8ee1f6ab730863bfad55bd
* maintenance: Migrate some Database::select() calls to SQBAmir Sarabadani2023-09-151-17/+12
| | | | | | | | | | | Done semi-automatically via migrateselect[1]. The script only accepted ascii chars until I found out and fixed it and now I can run it in more places. [1] https://gitlab.wikimedia.org/ladsgroup/migrateselect Bug: T344971 Change-Id: I83b6c424c62a517a0ab3635b64488ea53fd88bab
* Introduce RevisionSelectQueryBuilderAmir Sarabadani2023-09-061-10/+7
| | | | | | | | | Deprecating RevisionStore::getQueryInfo() and cleaning up a lot of code Also removing a brittle test that wasn't really testing anything. Bug: T344971 Change-Id: Ifd690dc8f030f86e3567a717eaeb830cb6dc703b
* maintenance: Begin using `Maintenance::getServiceContainer()`Derick Alangi2023-09-049-33/+23
| | | | | | | | | | | Maintenance class provides a method for getting a fresh reference of the MW services container instance. Let's make use of these in maintenance scripts now that we have it. NOTE: There are still some static methods like in refreshLinks.php that makes use of services that we can't use this method for now. Change-Id: Idba744057577896fc97c9ecf4724db27542bf01c
* compressOld: Do not assume the latest revision has the greatest IDFunc2023-08-051-6/+6
| | | | | | | Previous revisions can have bigger revision IDs than the last revision in a few situations, including imported or history-merged revisions. Change-Id: I4e23d10b0763de4b016460e789baec4f560f1674
* Merge "Migrate more calls of Database::select* to SelectQueryBuilder"jenkins-bot2023-07-261-6/+10
|\
| * Migrate more calls of Database::select* to SelectQueryBuilderAmir Sarabadani2023-07-261-6/+10
| | | | | | | | | | | | | | | | | | Using a php parser written on top of ANTLR4, done semi-automatically. I checked everything and made adjustments. Bug: T311866 Change-Id: I6150c6909bce8f3dbd745a26380cc0af9d9c547f
* | Simplify WHERE conditions with field IS NULLUmherirrender2023-07-242-2/+2
|/ | | | | | Reduce raw sql fragments on simple compares Change-Id: I3f2340dfdbf5197cc22546911e6c5653dc5a6269
* maintenance: Switch simple calls of Database::select to SQBAmir Sarabadani2023-07-198-157/+159
| | | | | | | Done semi-automatically via a php parser written on top of ANTLR4. Bug: T311866 Change-Id: I33f5b6703c0aa9c80c907a21c2a770e30642edd3
* Add maintenance/storage/fixLegacyEncoding.phpAmir Sarabadani2023-06-222-6/+78
| | | | | | | | | | | To fix legacy encoding entries in external storage which means they can't be fixed via calling moveToExternal.php. The script originally was copy-paste and clean up of moveToExternal.php but it made so much duplication that I went with subclassing. Bug: T282734 Change-Id: Ic52e843f3dbe7d14cc8df5e8f3fe7aada7681bc9
* Merge "Make some storage scripts use Maintenance class"jenkins-bot2023-06-163-103/+128
|\
| * Make some storage scripts use Maintenance classdaniel2023-06-143-103/+128
| | | | | | | | | | | | | | CommandLineInc is deprecated. This allows the scripts to be executed from MaintenanceRunner. Change-Id: I180605ea5cb47783670b28a6f01d98f0398c705d
* | moveToExternal: First decompress gziped entries before iconvAmir Sarabadani2023-06-151-0/+8
|/ | | | | | | This bug has messed up content of 0.5M revisions in English Wikipedia Bug: T128150 Change-Id: I675287a07a58df0f19a35011c012462400e90be8
* moveToExternal: Also check for utf8 encoding before trying to convertAmir Sarabadani2023-06-081-0/+1
| | | | | | | | | | | | | | | | | | | | | | | While most rows in production use 'utf-8' to flag content being UTF-8, we have lots of rows flagged with 'utf8': mysql:research@s3-analytics-replica.eqiad.wmnet [dawiki]> select old_flags, count(*) from text group by old_flags limit 50; +---------------------+----------+ | old_flags | count(*) | +---------------------+----------+ | error | 2 | | external,gzip | 49 | | external,object | 36 | | external,utf-8 | 1614469 | | external,utf8 | 336780 | | gzip,utf-8,external | 1094 | | utf-8,gzip,external | 9458083 | +---------------------+----------+ 7 rows in set (26.038 sec) This would confuse the script to try to reencode it again which possibly could lead to all sorts of errors Change-Id: I9b4a38538199c9954cfed51cdd2bba8b0f6cb953
* moveToExternal: Actually convert encoding of cur_textKevin Israel2023-06-031-1/+2
| | | | | | | | | HistoryBlobCurStub objects point to rows in the cur table from 1.4, which would have used the legacy encoding. It is incorrect to add the "utf-8" flag without also converting the encoding. Bug: T337700 Change-Id: Ie884512c1489358cabdf52660a7cb9d0797b8e78
* build: Cleanup of .phpcs.xmlUmherirrender2023-04-131-0/+1
| | | | | | Use inline suppression for known exception from eval/passthru/query call Change-Id: Ie85ea5698a615adf07e4e391bf06d102149effd5
* Fix some PHPStorm inspections (#1)Tim Starling2023-03-251-2/+2
| | | | | | | | | | | | * Triple backslash in regex should really be quadruple backslash * Using the returned value of a void method * Immediately overwritten array keys * Duplicate array keys * Foreach variable reuse * sprintf() with too many params * Incorrect reference usage Change-Id: I3c649b543c9561a1614058c50f3847f663ff04df
* Reorg: Namespace the Title classJames D. Forrester2023-03-023-0/+3
| | | | | | | | | | | | | | | | | | | This is moderately messy. Process was principally: * xargs rg --files-with-matches '^use Title;' | grep 'php$' | \ xargs -P 1 -n 1 sed -i -z 's/use Title;/use MediaWiki\\Title\\Title;/1' * rg --files-without-match 'MediaWiki\\Title\\Title;' . | grep 'php$' | \ xargs rg --files-with-matches 'Title\b' | \ xargs -P 1 -n 1 sed -i -z 's/\nuse /\nuse MediaWiki\\Title\\Title;\nuse /1' * composer fix Then manual fix-ups for a few files that don't have any use statements. Bug: T166010 Follows-Up: Ia5d8cb759dc3bc9e9bbe217d0fb109e2f8c4101a Change-Id: If8fc9d0d95fc1a114021e282a706fc3e7da3524b
* Reorg: Migrate WikiMap to WikiMap/ out of includesAmir Sarabadani2023-02-271-0/+1
| | | | | | | And WikiReference Bug: T321882 Change-Id: I60cf4b9ef02b9d58118caa39172677ddfe03d787
* Remove unused local variable assignmentUmherirrender2023-02-042-3/+0
| | | | | | Dead code found by phan Change-Id: I9fc404d546a4fb1c61394cb6359eb774fd94383a
* Merge "Update moveToExternal and resolveStubs"jenkins-bot2022-12-202-162/+367
|\
| * Update moveToExternal and resolveStubsTim Starling2022-12-202-162/+367
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Convert these two old scripts to Maintenance subclasses. * Uncomment the resolveStub() call in moveToExternal and fix one obvious bug with it, i.e. the fact that stubs need to be resolved after CGZ blobs are moved. * Replace get_class() with instanceof. * Make the "tiny text" threshold configurable. Normally this is not wanted in WMF production since new revisions are written to ES unconditionally. * Add a dry run mode. * Add an undo log. * Add --skip-resolve option. * Make resolveStub() be much more defensive about what it resolves. * In moveToExternal, make compression optional and do it also for plain text. * Optionally convert the legacy encoding to UTF-8. Bug: T299387 Change-Id: I52d54e3b6b785ac072796031be06499221340f51
generated by cgit v1.2.3 (git 2.25.1) at 2025-04-22 19:55:00 +0000