MWStake MediaWiki Manager: Difference between revisions

From MWStake
Jump to navigation Jump to search
No edit summary
Β 
(93 intermediate revisions by the same user not shown)
Line 1: Line 1:
Β  What is this about?
Β  '''[https://mwstake.org/mwstake/wiki/Special:AllPages?from=MWStake&to=&namespace=0 SUBPAGES]'''
Β  Β 
Β 
Β  1. Install MediaWiki.
Β  '''[https://github.com/dataspects/mediawiki-manager/search?q=CreateCampEMWCon2021 GitHub Repo Tag "CreateCampEMWCon2021"]'''
Β 
Code Base atΒ  Β  Β  Β  Β  Β  Β  https://github.com/dataspects/mediawiki-manager
Β  2. Manage (settings), upgrades, extensions, snapshots and content through an independent CLI/API/UI.
Development Discussion at https://riot.im/app/#/room/#mwdeployment:matrix.org
Β 
Β 
Β  3. Put MediaWiki in SafeMode in case of problems.
== Project Reasoning and Potential Goals ==
Β 
Facilitate managing all aspects and use cases regarding the operation of MediaWiki instance(s)
Β 
# Across set up modes and operating systems
# Targeting all user groups accordingly
# Conveying best practices
# Component monitoring and certification services
# Not integrated, but besides MediaWiki instance(s)
# Install/configure/upgrade system(s)
# Install/enable/disable/upgrade/configure extensions
#* MediaWiki Stakeholders Group Certified Extensions Service
# Backup/restore/clone/compare/consolidate system(s) (snapshots)
# Inject/extract/facet/edit/consolidate/import/export/remove apps/structures/ontologies
#* MediaWiki Stakeholders Group Certified Apps Service
# Script/automate/schedule tasks
# Package/move/archive/migrate system(s)
# ORTHOGONAL MODULARITY


__TOC__
__TOC__


<gallery mode="slideshow">
== Backlog ==
File:mediawiki-manager-Service-Architecture.png|MWM Service Architecture
File:Mediawiki-manager-InteractionLogics.png|MWM Interaction Logics
File:Mediawiki-manager-UI.png|MWM UI
File:Mediawiki-manager-ExtensionsStore.png|MWM Extensions Store
</gallery>


== Development Broadcasting ==
* Permissions (owner, group, rwx, executing user, etc.)
* Apache (SSL, configuration)
* Certified extensions and apps catalog (selection, fields, mechanisms, policies, stewardship)
* SSO
* Search


There are videos about MWStake MWM at [https://www.youtube.com/channel/UCoP-EgZJPhB0UE4WaKCHR1w dataspects' YouTube channel].
== MWM Implementation Aspects ==


* [https://www.youtube.com/watch?v=MTYq2a2Xsls First glimpse at the idea of an MWStake MediaWiki Manager]
'''How and Why?'''


== Target User Groups ==
* Dissect system&hellip;
* &hellip;into as few as possible but into as many as necessary interfacing components&hellip;
* &hellip;in order to fulfill all intended use cases&hellip;
* &hellip;in accordance with corresponding users' capabilities and needs.


=== UGcli: Users willing/able to use the command line ===
=== Abstraction Layers ===


==== UGclis: Users wanting to script actions ====
CLI <-- API <-- UI


=== UGui: Users requiring a UI, NOT willing/able to use the command line ===
# UI is GUI for API
# API wraps CLI
# [https://github.com/dataspects/mediawiki-manager/tree/main/cli CLI means bash scripts covering all use cases]


== Use Cases ==
=== [https://podman.io/ Podman]-managed [https://github.com/dataspects/mediawiki-manager/blob/main/mediawiki-manager.tpl MWM Kubernetes deploy manifest] ("Containers") ===
# Pod "mwm"
# [https://github.com/dataspects/dataspectsSystemBuilder/tree/master/docker-images/mediawiki Container "mediawiki"] + [https://github.com/dataspects/mediawiki-manager/blob/main/mediawiki-manager.tpl user volumes] and [https://github.com/dataspects/mediawiki-manager/tree/main/envs config envs]
# Container "mariadb" + config envs


=== Install MW System Instance ===
==== Config envs ====
Pending: [http://docs.podman.io/en/latest/markdown/podman-secret.1.html podman secret]
{|
|<syntaxhighlight lang=bash>
# my-new-system.env
export WG_DB_PASSWORD=8n65f%6n5
</syntaxhighlight>
|&rArr; envsubst &rArr;
|<syntaxhighlight lang=bash>
# mediawiki-manager.yml
...
containers:
Β  - image: docker.io/dataspects/mediawiki:1.35.0-2104121740
Β  Β  name: mediawiki
Β  Β  env:
Β  Β  Β  - name: WG_DB_PASSWORD
Β  Β  Β  Β  value: 8n65f%6n5
Β  Β  Β  - ...
...
</syntaxhighlight>
|&rArr;
|<syntaxhighlight lang=php># Localsettings.php
$wgDBpassword = getenv("WG_DB_PASSWORD");


=== Manage MW ===
</syntaxhighlight>
==== Settings ====
|}
# [https://github.com/dataspects/mediawiki-manager/blob/main/catalogues/systemSettings.json systemSettings.json]
==== Upgrade ====


# [https://github.com/dataspects/mediawiki-manager/blob/main/catalogues/versions.json upgrades.json]
==== Config database ====


{|
|<syntaxhighlight lang=php>
<?php
# addToMWMSQLite.php


=== Manage MW Extensions (install/enable/disable/upgrade) ===
$nameΒ  = $argv[1];
$localsettingsdirectives = $argv[2];
$db = new SQLite3('mwmconfigdb.sqlite');


# [https://github.com/dataspects/mediawiki-manager/blob/main/catalogues/extensions.json extensions.json]
$stmt = $db->prepare('INSERT INTO extensions (name, localsettingsdirectives) VALUES ( :name, :localsettingsdirectives)');
$stmt->bindValue(':name', $name, SQLITE3_TEXT);
$stmt->bindValue(':localsettingsdirectives', $localsettingsdirectives, SQLITE3_TEXT);


==== MediaWiki Stakeholders Group Certified Extensions Service ====
$stmt->execute();
</syntaxhighlight>
|&rArr;
|<syntaxhighlight lang=php>
<?php
# updateMWMLocalSettings.php


* Some background:
$db = new SQLite3('mwmconfigdb.sqlite');
** [[Action:7]]
$stmt = $db->prepare('SELECT localsettingsdirectives FROM extensions');
** [[Talk:MWStake_incorporation/Board_meeting]]
$result = $stmt->execute();
* Format right now: https://github.com/dataspects/mediawiki-manager/blob/main/catalogues/extensions.json
* Management, roles, responsibilities, privileges, processes, QA?
* Hosting?


=== Snapshots (backup/restore/clone) ===
$mwmLocalSettingsString = "";
while($res = $result->fetchArray(SQLITE3_ASSOC)){
Β  Β  $mwmLocalSettingsString .= trim($res["localsettingsdirectives"])."\n";
}


=== SafeMode ===
$mwmLS = fopen("mwmLocalSettings.php", "w");
fwrite($mwmLS, "<?php\n".$mwmLocalSettingsString);
fclose($mwmLS);
</syntaxhighlight>
|&rArr;
|<syntaxhighlight lang=php># Localsettings.php
$mwmls = "../mwmLocalSettings.php";
# MWStake MediaWiki Manager
if(file_exists($mwmls)) {
require_once($mwmls);
} else {
echo "ERROR: ".$mwmls." include not loaded.";
}
</syntaxhighlight>
|}


'''Idea''': if someone corrupts <code>extensions/</code> and/or <code>LocalSettings.php</code>, then <code>./start-SAFE-MODE.sh</code> restarts the wiki in safe mode, by falling back to container-internal versions of <code>extensions/</code> and <code>LocalSettings.php</code>.
== Target User Groups ==
<div style="margin:20px; padding:50px; border-radius:50px; background-color:beige;">
'''Who can use the concepts in MWM how?'''
Here I try to segregate target users into groups that share capabilities and needs.
{|class=wikitable
!User Group
!Point/Select/Type/Click
!CLI/SSH
!Scripting
!Edit configuration junior
!Edit configuration senior
|-
|''Examples''
|''"Web Interface"''
|
|''Automate/schedule tasks''
*''CLI: e.g. [https://github.com/dataspects/mediawiki-manager/blob/main/cli/manage-extensions/tests.sh manage extensions]''
*''API: (pending)''
|
*''Edit LocalSettings.php''
*''Run composer''
|
PLUS
*''Interpret logs''<br>
*''Report bugs/feature request''<br>
*''Branch code, develop and PR''
|-
|{{mo|'''UGUI'''}} "WebAdmin"
|Yes
| -
|"Macro" style only from within UI<br>("Excel Record Macro Style")
| -
| -
|-
|{{mo|'''UGAdmin'''}} "SysAdmin"
|Yes
|Yes
|Run/arrange .sh scripts
|Yes
| -
|-
|{{mo|'''UGCoder'''}} "Developer"
|Yes
|Yes
|Run/arrange/compose/create/edit .sh scripts including API calls
|Yes
|Yes
|}
</div>


[[File:MWMSafeModeAlert.png]]
== Use Cases ==
Β 
=== {{mlb|Install/configure/upgrade/run/operate system(s)}} ===
Β 
# Favored path: Kubernetes-based only for mwmITLocal, mwmITIntra and mwmITCloud


{|class=wikitable
{|class=wikitable
!MWM Normal Mode
!Where
!MWM Safe Mode
!How
!Why
|-
|-
|mwmITLocal: locally on a single computer
|containerized
|for personal/private/development use
|-
|mwmITIntra: in an intranet
|
|
[[File:mwm-normal-mode.png]]
containerized
https://github.com/dataspects/mediawiki-manager/blob/main/start.sh
|for corporate internal use
<syntaxhighlight lang=yaml>
|-
version: "3.7"
|mwmITCloud: on the internet/in the cloud
services:
|containerized
Β  mediawiki:
|for corporate internal/external/public use
Β  Β  env_file:
Β  Β  Β  - CanastaInstanceSettings.env
Β  Β  container_name: mediawiki_canasta
Β  Β  image: dataspects/mediawiki:1.35.0-2103040820
Β  Β  ports:
Β  Β  Β  - 80:80
Β  Β  Β  - 443:443
Β  Β  volumes:
Β  Β  Β  # MediaWiki >>>
Β  Β  Β  - ./mediawiki_root/w/LocalSettings.php:/var/www/html/w/LocalSettings.php
Β  Β  Β  - ./mediawiki_root/w/extensions:/var/www/html/w/extensions
Β  Β  Β  - ./mediawiki_root/w/skins:/var/www/html/w/skins
Β  Β  Β  - ./mediawiki_root/w/vendor:/var/www/html/w/vendor
Β  Β  Β  - ./mediawiki_root/w/images:/var/www/html/w/images
Β  Β  Β  # MediaWiki Manager >>>
Β  Β  Β  - ./mediawiki_root/api:/var/www/html/api
Β  Β  Β  - ./mediawiki_root/dsmwm.log:/var/www/html/dsmwm.log
Β  Β  Β  - ./mediawiki_root/restic_password:/var/www/html/restic_password
Β  Β  Β  - ./mediawiki_root/ui:/var/www/html/ui
Β  Β  Β  - ./mediawiki_root/cloneLocation:/var/www/html/cloneLocation
Β  Β  Β  - ./restic_data:/var/www/html/restic-repo
Β  Β  Β  - ./conf:/etc/apache2/sites-available
Β  Β  # https://phabricator.wikimedia.org/source/mediawiki/browse/master/docker-compose.yml
Β  Β  # https://phoenixnap.com/kb/how-to-share-data-between-docker-containers
Β  Β  # + mysql
Β  # mwmapi:
Β  #Β  container_name: mwmapi
Β  #Β  image: dataspects/mwmapi:210303
Β  mysql:
Β  Β  container_name: mysql_canasta
Β  Β  image: mariadb:10.5.5
Β  Β  volumes:
Β  Β  Β  - ./mysql_data:/var/lib/mysql
Β  Β  environment:
Β  Β  Β  - MYSQL_ROOT_PASSWORD=${MARIADB_ROOT_PASSWORD}
</syntaxhighlight>
|
[[File:mwm-safe-mode.png]]
https://github.com/dataspects/mediawiki-manager/blob/main/start-SAFE-MODE.sh
<syntaxhighlight lang=yaml>
version: "3.7"
services:
Β  mediawiki:
Β  Β  env_file:
Β  Β  Β  - CanastaInstanceSettings.env
Β  Β  container_name: mediawiki_canasta
Β  Β  image: dataspects/mediawiki:1.35.0-2103040820
Β  Β  ports:
Β  Β  Β  - 80:80
Β  Β  Β  - 443:443
Β  Β  volumes:
Β  Β  Β  # MediaWiki >>>
Β  Β  Β  #- ./mediawiki_root/w/LocalSettings.php:/var/www/html/w/LocalSettings.php
Β  Β  Β  #- ./mediawiki_root/w/extensions:/var/www/html/w/extensions
Β  Β  Β  #- ./mediawiki_root/w/skins:/var/www/html/w/skins
Β  Β  Β  #- ./mediawiki_root/w/vendor:/var/www/html/w/vendor
Β  Β  Β  - ./mediawiki_root/w/images:/var/www/html/w/images
Β  Β  Β  # MediaWiki Manager >>>
Β  Β  Β  - ./mediawiki_root/api:/var/www/html/api
Β  Β  Β  - ./mediawiki_root/dsmwm.log:/var/www/html/dsmwm.log
Β  Β  Β  - ./mediawiki_root/restic_password:/var/www/html/restic_password
Β  Β  Β  - ./mediawiki_root/ui:/var/www/html/ui
Β  Β  Β  - ./mediawiki_root/cloneLocation:/var/www/html/cloneLocation
Β  Β  Β  - ./restic_data:/var/www/html/restic-repo
Β  Β  Β  - ./conf:/etc/apache2/sites-available
Β  Β  # https://phabricator.wikimedia.org/source/mediawiki/browse/master/docker-compose.yml
Β  Β  # https://phoenixnap.com/kb/how-to-share-data-between-docker-containers
Β  Β  # + mysql
Β  # mwmapi:
Β  #Β  container_name: mwmapi
Β  #Β  image: dataspects/mwmapi:210303
Β  mysql:
Β  Β  container_name: mysql_canasta
Β  Β  image: mariadb:10.5.5
Β  Β  volumes:
Β  Β  Β  - ./mysql_data:/var/lib/mysql
Β  Β  environment:
Β  Β  Β  - MYSQL_ROOT_PASSWORD=${MARIADB_ROOT_PASSWORD}
</syntaxhighlight>
|}
|}


=== Manage MW Content (inject/extract/facet/edit) ===
=== {{mlb|Install/enable/disable/upgrade/configure extensions}} ===
Β 
[[MWStake MediaWiki Manager/MWStakeServices/ExtensionsCatalog]]
Β 
=== {{mlb|Backup/restore/clone/compare/consolidate system(s) (snapshots)}} ===
Β 
=== {{mlb|Inject/extract/facet/edit/consolidate/import/export/remove apps/structures/ontologies}} ===
Β 
==== CLI (Bash) ====
Β 
https://github.com/dataspects/mediawiki-manager/tree/main/cli/manage-content
Β 
=== {{mlb|Scripting}} ===
Β 
==== CLI (Bash) ====
Β 
https://github.com/dataspects/mediawiki-manager/blob/main/cli/manage-extensions/tests.sh
Β 
=== {{mlb|Package system(s)}} ===


# [https://github.com/dataspects/mediawiki-manager/blob/main/catalogues/apps.json mediawiki-apps.json]
# [https://github.com/dataspects/mediawiki-manager/blob/main/catalogues/apps.json mediawiki-apps.json]
== MWStake MWM Service Portfolio ==
[[MWStake_MediaWiki_Manager/MWStakeServices]]


== Abstraction Layers ==
== Abstraction Layers ==


=== ALcontainerization: Docker ===
[[File:Mediawiki-manager-Service-Architecture.png|1000px]]


# The [https://hub.docker.com/r/dataspects/mediawiki/tags?page=1&ordering=last_updated mediawiki docker image] contains a [https://github.com/dataspects/dataspectsSystemBuilder/blob/master/docker-images/mediawiki/Dockerfile full MediaWiki installation].
=== ALcontainerization: Podman ===
# [https://github.com/dataspects/mediawiki-manager/blob/main/cli/install-system/initialize-persistent-mediawiki-service-volumes.sh initialize-persistent-mediawiki-service-volumes.sh] will copy the 5 persistence-relevant files/directories out to the host so they can be volumed in when starting the mediawiki container.
Β 
# This shall allow for [https://github.com/dataspects/mediawiki-manager/blob/main/cli/manage-system/start-SAFE-MODE.sh start-SAFE-MODE.sh].
# The [https://hub.docker.com/r/dataspects/mediawiki/tags?page=1&ordering=last_updated mediawiki container image] contains a [https://github.com/dataspects/dataspectsSystemBuilder/blob/master/docker-images/mediawiki/Dockerfile full MediaWiki installation].
# [https://github.com/dataspects/mediawiki-manager/blob/main/cli/install-system/initialize-persistent-mediawiki-service-volumes.sh initialize-persistent-mediawiki-service-volumes.sh] will copy persistence-relevant files/directories out to the host so they can be volumed in when starting the mediawiki container.


=== ALcli: shell scripts using MWAPI and CRUDing files in service volumes ===
=== ALcli: shell scripts using MWAPI and CRUDing files in service volumes ===


https://github.com/dataspects/mediawiki-manager
=== ALapi: MWM API wrapping and parametrizing ALcli shell scripts and using MWAPI ===


# [https://github.com/dataspects/mediawiki-manager/blob/main/docker-compose.yml docker-compose.yml] ([https://github.com/dataspects/dataspectsSystemBuilder/tree/master/docker-images/mediawiki MediaWiki Dockerfile])
==== Unit Testing ====


=== ALapi: MWM API (Go) wrapping and parametrizing ALcli shell scripts ===
=== ALui: MWM UI using and parametrizing ALapi endpoints ===


https://github.com/dataspects/mwmapi
[[File:Mediawiki-manager-UI.png|MWM UI]]


==== Unit Testing ====
==== Integration Testing ====


=== ALui: MWM UI (React/Material UI) using and parametrizing ALapi endpoints ===
== Development Broadcasting ==


https://github.com/dataspects/mwmui
There are videos about MWStake MWM at [https://www.youtube.com/channel/UCoP-EgZJPhB0UE4WaKCHR1w dataspects' YouTube channel].


==== Integration Testing ====
* [https://www.youtube.com/watch?v=MTYq2a2Xsls First glimpse at the idea of an MWStake MediaWiki Manager]

Latest revision as of 08:20, 22 April 2021

SUBPAGES
GitHub Repo Tag "CreateCampEMWCon2021"
Code Base at              https://github.com/dataspects/mediawiki-manager
Development Discussion at https://riot.im/app/#/room/#mwdeployment:matrix.org

Project Reasoning and Potential Goals

Facilitate managing all aspects and use cases regarding the operation of MediaWiki instance(s)
  1. Across set up modes and operating systems
  2. Targeting all user groups accordingly
  3. Conveying best practices
  4. Component monitoring and certification services
  5. Not integrated, but besides MediaWiki instance(s)
  6. Install/configure/upgrade system(s)
  7. Install/enable/disable/upgrade/configure extensions
    • MediaWiki Stakeholders Group Certified Extensions Service
  8. Backup/restore/clone/compare/consolidate system(s) (snapshots)
  9. Inject/extract/facet/edit/consolidate/import/export/remove apps/structures/ontologies
    • MediaWiki Stakeholders Group Certified Apps Service
  10. Script/automate/schedule tasks
  11. Package/move/archive/migrate system(s)
  12. ORTHOGONAL MODULARITY

Backlog

  • Permissions (owner, group, rwx, executing user, etc.)
  • Apache (SSL, configuration)
  • Certified extensions and apps catalog (selection, fields, mechanisms, policies, stewardship)
  • SSO
  • Search

MWM Implementation Aspects

How and Why?

  • Dissect system…
  • …into as few as possible but into as many as necessary interfacing components…
  • …in order to fulfill all intended use cases…
  • …in accordance with corresponding users' capabilities and needs.

Abstraction Layers

CLI <-- API <-- UI
  1. UI is GUI for API
  2. API wraps CLI
  3. CLI means bash scripts covering all use cases

Podman-managed MWM Kubernetes deploy manifest ("Containers")

  1. Pod "mwm"
  2. Container "mediawiki" + user volumes and config envs
  3. Container "mariadb" + config envs

Config envs

Pending: podman secret

# my-new-system.env
export WG_DB_PASSWORD=8n65f%6n5
⇒ envsubst ⇒
# mediawiki-manager.yml
...
containers:
  - image: docker.io/dataspects/mediawiki:1.35.0-2104121740
    name: mediawiki
    env:
      - name: WG_DB_PASSWORD
        value: 8n65f%6n5
      - ...
...
# Localsettings.php
$wgDBpassword = getenv("WG_DB_PASSWORD");

Config database

<?php
# addToMWMSQLite.php

$name  = $argv[1];
$localsettingsdirectives = $argv[2];
$db = new SQLite3('mwmconfigdb.sqlite');

$stmt = $db->prepare('INSERT INTO extensions (name, localsettingsdirectives) VALUES ( :name, :localsettingsdirectives)');
$stmt->bindValue(':name', $name, SQLITE3_TEXT);
$stmt->bindValue(':localsettingsdirectives', $localsettingsdirectives, SQLITE3_TEXT);

$stmt->execute();
<?php
# updateMWMLocalSettings.php

$db = new SQLite3('mwmconfigdb.sqlite');
$stmt = $db->prepare('SELECT localsettingsdirectives FROM extensions');
$result = $stmt->execute();

$mwmLocalSettingsString = "";
while($res = $result->fetchArray(SQLITE3_ASSOC)){
    $mwmLocalSettingsString .= trim($res["localsettingsdirectives"])."\n";
}

$mwmLS = fopen("mwmLocalSettings.php", "w");
fwrite($mwmLS, "<?php\n".$mwmLocalSettingsString);
fclose($mwmLS);
# Localsettings.php
$mwmls = "../mwmLocalSettings.php";
# MWStake MediaWiki Manager
if(file_exists($mwmls)) {
	require_once($mwmls);
} else { 
	echo "ERROR: ".$mwmls." include not loaded.";
}

Target User Groups

Who can use the concepts in MWM how? Here I try to segregate target users into groups that share capabilities and needs.

User Group Point/Select/Type/Click CLI/SSH Scripting Edit configuration junior Edit configuration senior
Examples "Web Interface" Automate/schedule tasks
  • Edit LocalSettings.php
  • Run composer

PLUS

  • Interpret logs
  • Report bugs/feature request
  • Branch code, develop and PR
UGUI "WebAdmin" Yes - "Macro" style only from within UI
("Excel Record Macro Style")
- -
UGAdmin "SysAdmin" Yes Yes Run/arrange .sh scripts Yes -
UGCoder "Developer" Yes Yes Run/arrange/compose/create/edit .sh scripts including API calls Yes Yes

Use Cases

Install/configure/upgrade/run/operate system(s)

  1. Favored path: Kubernetes-based only for mwmITLocal, mwmITIntra and mwmITCloud
Where How Why
mwmITLocal: locally on a single computer containerized for personal/private/development use
mwmITIntra: in an intranet

containerized

for corporate internal use
mwmITCloud: on the internet/in the cloud containerized for corporate internal/external/public use

Install/enable/disable/upgrade/configure extensions

MWStake MediaWiki Manager/MWStakeServices/ExtensionsCatalog

Backup/restore/clone/compare/consolidate system(s) (snapshots)

Inject/extract/facet/edit/consolidate/import/export/remove apps/structures/ontologies

CLI (Bash)

https://github.com/dataspects/mediawiki-manager/tree/main/cli/manage-content

Scripting

CLI (Bash)

https://github.com/dataspects/mediawiki-manager/blob/main/cli/manage-extensions/tests.sh

Package system(s)

  1. mediawiki-apps.json

MWStake MWM Service Portfolio

MWStake_MediaWiki_Manager/MWStakeServices

Abstraction Layers

Mediawiki-manager-Service-Architecture.png

ALcontainerization: Podman

  1. The mediawiki container image contains a full MediaWiki installation.
  2. initialize-persistent-mediawiki-service-volumes.sh will copy persistence-relevant files/directories out to the host so they can be volumed in when starting the mediawiki container.

ALcli: shell scripts using MWAPI and CRUDing files in service volumes

ALapi: MWM API wrapping and parametrizing ALcli shell scripts and using MWAPI

Unit Testing

ALui: MWM UI using and parametrizing ALapi endpoints

MWM UI

Integration Testing

Development Broadcasting

There are videos about MWStake MWM at dataspects' YouTube channel.