Project - SemanticInformationRetrieval (SIR) - Revision history

Lex Sulzer: /* Preamble */

2018-02-02T15:49:14Z

Preamble

← Older revision		Revision as of 11:49, 2 February 2018
Line 16:		Line 16:

	''Write the code you wish you had.''		''Write the code you wish you had.''

			== Example ==

			* Project page: https://dataspects.com/dataspects-search
			* Development version: https://dataspects.com/search

	== Intention/Goal/Purpose/Fundamentals (Current Thoughts)==		== Intention/Goal/Purpose/Fundamentals (Current Thoughts)==

Lex Sulzer: /* Intention/Goal/Purpose/Fundamentals (Current Thoughts) */

2017-10-08T16:33:21Z

Intention/Goal/Purpose/Fundamentals (Current Thoughts)

← Older revision		Revision as of 12:33, 8 October 2017
Line 31:		Line 31:
	# SIR does represent several designated MediaWiki extensions:		# SIR does represent several designated MediaWiki extensions:
	#* '''Extension:SIRIndexer''' providing a DSL for '''mapping and indexing''' both a page's '''factorized information in template instances''' as well as its '''free text'''.		#* '''Extension:SIRIndexer''' providing a DSL for '''mapping and indexing''' both a page's '''factorized information in template instances''' as well as its '''free text'''.
	#* '''Extension:~~SIRSearchInterface~~''' providing an API for '''search interfaces''' on '''special pages''' and by '''parser functions'''.		#* '''Extension:SIRUserInterface''' providing an API for '''search interfaces''' on '''special pages''' and by '''parser functions'''.
	#* '''Extension:~~SIRElasticSearchBackend~~''' providing fast "ES cached" page wikitext HTML delivery.		#* '''Extension:SIRBackend''' providing fast "ES cached" page wikitext HTML delivery.
	#** Such search interfaces shall be easily '''configured for covering any arbitrary facet of any complexity at the outset''', i.e. like "searching in a namespace" but extended to "searching in a facet".		#** Such search interfaces shall be easily '''configured for covering any arbitrary facet of any complexity at the outset''', i.e. like "searching in a namespace" but extended to "searching in a facet".
	# Truly [https://smw-cindykate.com/main/Component_251423 relevant full-text search] cannot be achieved without tuning the analysis process both at index (defining features) and query (extracting signals) time. That's why '''SIR provides DSL methods''' that		# Truly [https://smw-cindykate.com/main/Component_251423 relevant full-text search] cannot be achieved without tuning the analysis process both at index (defining features) and query (extracting signals) time. That's why '''SIR provides DSL methods''' that

Lex Sulzer: /* Architectural Ideas (work in progress) */

2017-10-08T16:28:12Z

Architectural Ideas (work in progress)

← Older revision		Revision as of 12:28, 8 October 2017
Line 43:		Line 43:
	[[File:SDMS_Architecture.png]]		[[File:SDMS_Architecture.png]]

	* When delivering a page, the web server assembles the response by normal MW functionality (caching!) but keeps a placeholder where the page's wikitext HTML would normally go. ([https://www.mediawiki.org/wiki/Manual:Article.php ~~Article~~::view])		* When delivering a page, the web server assembles the response by normal MW functionality (caching!) but keeps a placeholder where the page's wikitext HTML would normally go. (https://www.mediawiki.org/wiki/Manual:Article.php ::view, would https://www.mediawiki.org/wiki/Manual:Hooks/ArticleFromTitle be an appropriate hook?)
	** The page's HTML is obtained from the ElasticSearch server.		** The page's HTML is obtained from the ElasticSearch server.
	** Extension:SDMS has a configuration setting to override normal MW page delivery functionality accordingly.		** Extension:SDMS has a configuration setting to override normal MW page delivery functionality accordingly.

Lex Sulzer: /* Architectural Ideas (work in progress) */

2017-10-08T16:24:55Z

Architectural Ideas (work in progress)

← Older revision		Revision as of 12:24, 8 October 2017
Line 43:		Line 43:
	[[File:SDMS_Architecture.png]]		[[File:SDMS_Architecture.png]]

	* When delivering a page, the web server assembles the response by normal MW functionality (caching!) but keeps a placeholder where the page's wikitext HTML would normally go.		* When delivering a page, the web server assembles the response by normal MW functionality (caching!) but keeps a placeholder where the page's wikitext HTML would normally go. ([https://www.mediawiki.org/wiki/Manual:Article.php Article::view])
	** The page's HTML is obtained from the ElasticSearch server.		** The page's HTML is obtained from the ElasticSearch server.
	** Extension:SDMS has a configuration setting to override normal MW page delivery functionality accordingly.		** Extension:SDMS has a configuration setting to override normal MW page delivery functionality accordingly.

Lex Sulzer: /* Intention/Goal/Purpose/Fundamentals (Current Thoughts) */

2017-10-08T14:16:58Z

Intention/Goal/Purpose/Fundamentals (Current Thoughts)

← Older revision		Revision as of 10:16, 8 October 2017
Line 29:		Line 29:
	# SIR provides its DSL '''one abstraction level above MediaWiki's API''', as from an organization's perspective, SMW is one component of its comprehensive knowledge management system (albeit it might be its central component).		# SIR provides its DSL '''one abstraction level above MediaWiki's API''', as from an organization's perspective, SMW is one component of its comprehensive knowledge management system (albeit it might be its central component).
	# SIR provides '''versatile polymorphic search interfaces''' for '''full-text search''' as well as (subsequent) '''facetted interactive search''' along '''[https://everypageispageone.com/tag/subject-affinity/ subject affinities] paths through semanticized information''' ([http://everypageispageone.com/the-book/ EPPO-style knowledge topics]).		# SIR provides '''versatile polymorphic search interfaces''' for '''full-text search''' as well as (subsequent) '''facetted interactive search''' along '''[https://everypageispageone.com/tag/subject-affinity/ subject affinities] paths through semanticized information''' ([http://everypageispageone.com/the-book/ EPPO-style knowledge topics]).
	# SIR does ~~not~~ represent ~~a single~~ designated MediaWiki ~~extension, but provides two independent MW~~ extensions:		# SIR does represent several designated MediaWiki extensions:
	#* '''Extension:SIRIndexer''' providing an DSL for '''mapping and indexing''' both a page's '''factorized information in template instances''' as well as its '''free text'''.		#* '''Extension:SIRIndexer''' providing a DSL for '''mapping and indexing''' both a page's '''factorized information in template instances''' as well as its '''free text'''.
	#* '''Extension:SIRSearchInterface''' providing an API for '''search interfaces''' on '''special pages''' and by '''parser functions'''.		#* '''Extension:SIRSearchInterface''' providing an API for '''search interfaces''' on '''special pages''' and by '''parser functions'''.
			#* '''Extension:SIRElasticSearchBackend''' providing fast "ES cached" page wikitext HTML delivery.
	#** Such search interfaces shall be easily '''configured for covering any arbitrary facet of any complexity at the outset''', i.e. like "searching in a namespace" but extended to "searching in a facet".		#** Such search interfaces shall be easily '''configured for covering any arbitrary facet of any complexity at the outset''', i.e. like "searching in a namespace" but extended to "searching in a facet".
	# Truly [https://smw-cindykate.com/main/Component_251423 relevant full-text search] cannot be achieved without tuning the analysis process both at index (defining features) and query (extracting signals) time. That's why '''SIR provides DSL methods''' that		# Truly [https://smw-cindykate.com/main/Component_251423 relevant full-text search] cannot be achieved without tuning the analysis process both at index (defining features) and query (extracting signals) time. That's why '''SIR provides DSL methods''' that

Lex Sulzer at 14:11, 8 October 2017

2017-10-08T14:11:55Z

← Older revision		Revision as of 10:11, 8 October 2017
Line 1:		Line 1:
			__TOC__

	''Notes'':		''Notes'':
	* This project is suggested based on a discussion held at MWStake's August 2017 monthly meeting (see line 113 on https://etherpad.wikimedia.org/p/mwstake-2017-08).		* This project is suggested based on a discussion held at MWStake's August 2017 monthly meeting (see line 113 on https://etherpad.wikimedia.org/p/mwstake-2017-08).

Lex Sulzer: /* Intention/Goal/Purpose/Fundamentals (Current Thoughts) */

2017-10-08T14:09:08Z

Intention/Goal/Purpose/Fundamentals (Current Thoughts)

← Older revision		Revision as of 10:09, 8 October 2017
Line 35:		Line 35:
	#* are '''sufficiently high-level/concrete''', so that '''improving search relevance''' for a customer's SMW becomes straightforward enough '''without low-level coding''' — e.g. adding and composing document fields (SIR DSL) and their analysis (e.g. Elasticsearch JSON input).		#* are '''sufficiently high-level/concrete''', so that '''improving search relevance''' for a customer's SMW becomes straightforward enough '''without low-level coding''' — e.g. adding and composing document fields (SIR DSL) and their analysis (e.g. Elasticsearch JSON input).
	# SIR can be morphed into (S)MW's single source of truth, i.e. '''replacing its SQL backend'''.		# SIR can be morphed into (S)MW's single source of truth, i.e. '''replacing its SQL backend'''.

			== Architectural Ideas (work in progress) ==

			[[File:SDMS_Architecture.png]]

			* When delivering a page, the web server assembles the response by normal MW functionality (caching!) but keeps a placeholder where the page's wikitext HTML would normally go.
			** The page's HTML is obtained from the ElasticSearch server.
			** Extension:SDMS has a configuration setting to override normal MW page delivery functionality accordingly.
			* When delivering a page in edit mode, no normal MW functionality is changed.
			** When receiving an edited page:
			**# Execute normal MW page store procedure.
			**# Update corresponding ElasticSearch page document with:
			*# Non-analyzed raw HTML by parsing page wikitext into HTML
			*# Full-text analyzed HTML
			*# Structured page data by analyzing template calls (direct annotations are ignored as anti-best practice).

Lex Sulzer: /* Intention/Goal/Purpose/Fundamentals (Current Thoughts) */

2017-10-04T12:13:45Z

Intention/Goal/Purpose/Fundamentals (Current Thoughts)

← Older revision		Revision as of 08:13, 4 October 2017
Line 26:		Line 26:

	# SIR provides its DSL '''one abstraction level above MediaWiki's API''', as from an organization's perspective, SMW is one component of its comprehensive knowledge management system (albeit it might be its central component).		# SIR provides its DSL '''one abstraction level above MediaWiki's API''', as from an organization's perspective, SMW is one component of its comprehensive knowledge management system (albeit it might be its central component).
	# SIR provides '''versatile polymorphic search interfaces''' for '''full-text search''' as well as (subsequent) '''facetted interactive search''' along '''[https://everypageispageone.com/tag/subject-affinity/ subject affinities] paths through semanticized information'''.		# SIR provides '''versatile polymorphic search interfaces''' for '''full-text search''' as well as (subsequent) '''facetted interactive search''' along '''[https://everypageispageone.com/tag/subject-affinity/ subject affinities] paths through semanticized information''' ([http://everypageispageone.com/the-book/ EPPO-style knowledge topics]).
	# SIR does not represent a single designated MediaWiki extension, but provides two independent MW extensions:		# SIR does not represent a single designated MediaWiki extension, but provides two independent MW extensions:
	#* '''Extension:SIRIndexer''' providing an DSL for '''mapping and indexing''' both a page's '''factorized information in template instances''' as well as its '''free text'''.		#* '''Extension:SIRIndexer''' providing an DSL for '''mapping and indexing''' both a page's '''factorized information in template instances''' as well as its '''free text'''.
Line 34:		Line 34:
	#* e.g. in the case of Elasticsearch, take '''raw JSON as an input''', and		#* e.g. in the case of Elasticsearch, take '''raw JSON as an input''', and
	#* are '''sufficiently high-level/concrete''', so that '''improving search relevance''' for a customer's SMW becomes straightforward enough '''without low-level coding''' — e.g. adding and composing document fields (SIR DSL) and their analysis (e.g. Elasticsearch JSON input).		#* are '''sufficiently high-level/concrete''', so that '''improving search relevance''' for a customer's SMW becomes straightforward enough '''without low-level coding''' — e.g. adding and composing document fields (SIR DSL) and their analysis (e.g. Elasticsearch JSON input).
			# SIR can be morphed into (S)MW's single source of truth, i.e. '''replacing its SQL backend'''.

Lex Sulzer at 11:17, 4 October 2017

2017-10-04T11:17:31Z

← Older revision		Revision as of 07:17, 4 October 2017
Line 3:		Line 3:
	* It is the subject of a workshop at the upcoming SMWCon Fall 2017: [https://www.semantic-mediawiki.org/wiki/SMWCon_Fall_2017/Taking_the_first_step_towards_a_Semantic_Information_Retrieval_System_(SIRS) SMWCon_Fall_2017/Taking the first step towards a Semantic Information Retrieval System (SIRS)].		* It is the subject of a workshop at the upcoming SMWCon Fall 2017: [https://www.semantic-mediawiki.org/wiki/SMWCon_Fall_2017/Taking_the_first_step_towards_a_Semantic_Information_Retrieval_System_(SIRS) SMWCon_Fall_2017/Taking the first step towards a Semantic Information Retrieval System (SIRS)].
	* This is a working page representing consolidated current views. It is therefore subject to frequent changes.		* This is a working page representing consolidated current views. It is therefore subject to frequent changes.
	** Preliminary non-coherent prototype code snippets can be found at https://github.com/MWStake/SemanticInformationRetrieval. They were extracted from my experimental updated-upon-save/API-edit and structured-data-oriented [https://smw-cindykate.com/main/Community:Search\|Keyword Search].		** Preliminary non-coherent prototype code snippets can be found at https://github.com/MWStake/SemanticInformationRetrieval. They were extracted from my experimental updated-upon-save/API-edit and structured-data-oriented [https://smw-cindykate.com/main/Community:Search Keyword Search].

	== Preamble ==		== Preamble ==

Lex Sulzer: Corrected typo

2017-09-26T07:07:09Z

Corrected typo

← Older revision		Revision as of 03:07, 26 September 2017
Line 29:		Line 29:
	# SIR does not represent a single designated MediaWiki extension, but provides two independent MW extensions:		# SIR does not represent a single designated MediaWiki extension, but provides two independent MW extensions:
	#* '''Extension:SIRIndexer''' providing an DSL for '''mapping and indexing''' both a page's '''factorized information in template instances''' as well as its '''free text'''.		#* '''Extension:SIRIndexer''' providing an DSL for '''mapping and indexing''' both a page's '''factorized information in template instances''' as well as its '''free text'''.
	#* '''Extension:SIRSearchInterface''' providing an API for '''search interfaces''' on a '''special pages''' and by '''parser functions'''.		#* '''Extension:SIRSearchInterface''' providing an API for '''search interfaces''' on '''special pages''' and by '''parser functions'''.
	#** Such search interfaces shall be easily '''configured for covering any arbitrary facet of any complexity at the outset''', i.e. like "searching in a namespace" but extended to "searching in a facet".		#** Such search interfaces shall be easily '''configured for covering any arbitrary facet of any complexity at the outset''', i.e. like "searching in a namespace" but extended to "searching in a facet".
	# Truly [https://smw-cindykate.com/main/Component_251423 relevant full-text search] cannot be achieved without tuning the analysis process both at index (defining features) and query (extracting signals) time. That's why '''SIR provides DSL methods''' that		# Truly [https://smw-cindykate.com/main/Component_251423 relevant full-text search] cannot be achieved without tuning the analysis process both at index (defining features) and query (extracting signals) time. That's why '''SIR provides DSL methods''' that
	#* e.g. in the case of Elasticsearch, take '''raw JSON as an input''', and		#* e.g. in the case of Elasticsearch, take '''raw JSON as an input''', and
	#* are '''sufficiently high-level/concrete''', so that '''improving search relevance''' for a customer's SMW becomes straightforward enough '''without low-level coding''' — e.g. adding and composing document fields (SIR DSL) and their analysis (e.g. Elasticsearch JSON input).		#* are '''sufficiently high-level/concrete''', so that '''improving search relevance''' for a customer's SMW becomes straightforward enough '''without low-level coding''' — e.g. adding and composing document fields (SIR DSL) and their analysis (e.g. Elasticsearch JSON input).