Talk:WikiDB

From TestWiki
Revision as of 12:09, 30 September 2008 by Msa11usec (Talk | contribs) (Some display suggestions: minor typo)

Jump to: navigation, search

Some old discussion has been deleted. See here for diffs: [1].

External queries

This extension is really cool. Do you have any plans concerning external (read-only) queries to the database? --Martin 23:51, 10 July 2006 (BST)

Can you please expand on this - I'm not sure what you mean. --HappyDog 23:55, 10 July 2006 (BST)
Sure. Suppose I have wiki site which is primarily a user-editable database. Each wiki page may provide several database records (let's say information on sequences of proteins of a given family). There is some text which gives some information about the link between these database entries, that's the traditional wiki part. This cannot be put into the database because it's changing often, it may become large, and it doesn't have a meaningful format--the only field name which would be appropriate would be "comments". But the database records are essential for let's say the community of molecular biologists, who use client software to query and analyze the protein from the database. There may be wikis which want to use the database records as illustration, or maybe group the database records differently and comment on them. The idea is to not duplicate the database, but use feeds which would be updated and propagated from the source. My point of view comes from what I did for inserting bibliographic records (PMIDs from Medline and ISBNs from isbndb.org), in wikis: see Biblio extension. Someone also pointed out the Wikicite project. I don't know where these folks are going exactly, but the idea is to build a database of bibliographic references. Have a look. --Martin 00:19, 11 July 2006 (BST)
I haven't really given this any thought yet, and I will take a look at the links shortly, but on the surface what you say sounds quite plausible. How I would implement this is to use a new action argument, e.g. $action=rawdata (or an existing action if a suitable one exists - raw?). So if you entered e.g. http://some.domain.org/w/index.php?title=Table:Companies&action=rawdata, you would get the data (and schema?) from the Companies table in a standard format, probably XML. Does this sound like it would do the trick? (Bear in mind that if the table definition is altered, the XML output will change) --HappyDog 00:58, 11 July 2006 (BST)
I was thinking of something like an SQL SELECT statement. More specifically, the only features that I think would be useful would be:
  • download the whole database for local use as you just mentioned.
  • select one or several records by giving one column name and passing a list of strings. All records for which one of those strings matches is selected.
I don't know if you can do this simply using MediaWiki. Basically the URL I would pass would be like http://some.domain.org/w/index.php?title=Table:Companies&action=rawdata&select_id=12,34,567 where id is a column name and 12, 34 and 567 are acceptable id values. Maybe it is easier to just write an independent PHP script which queries the database according to the CGI arguments which are found. --Martin 01:40, 11 July 2006 (BST)
I think that level of detail is outside the scope of v1, however you're right, it would be fairly trivial to write a PHP script to parse and manipulate the raw data dump, particularly if it is in a standard format such as XML (or, it just occurs to me, SQL - I'm sure that would be possible, and then you could simply import it straight into a traditional database... hmmm... interesting thought!) --HappyDog 02:45, 11 July 2006 (BST)

Count of records

Is it possible to get number of record definitions without listing all? --Neptus 21:34, 14 March 2007 (GMT)

Can you please elaborate. Do you mean via a tag (so it can be displayed in a page) or on the table definition. I'm sure it would be fairly easy to add in either case - please tell me what you would like to see and where, and I'll see what I can do. --HappyDog 23:15, 14 March 2007 (GMT)
I have 4000 definitions in my WikiDB dictonary now and it grows all the time. I can count it only using tag <repeat table...># ... </repeat> now. This listing is long.
It will be nice to have only one aktual number of positions declared in WikiDB on page (or a number of selected records), visual effect like a number of pages {{NUMBEROFARTICLES}} in wiki - simple base counter: how many records is inside. Nice gadget for information and for a boast. Then it will be a tag, I think. --Neptus 01:40, 17 March 2007 (GMT)
The problem is that it would be a single number for the whole wiki. If I added a {{NUMBEROFROWS}} magic word it would give the no. of rows in all tables in the wiki, combined. It might be possible to somehow supply an argument to a magic word (e.g. {{NUMBEROFROWS|Table:Companies}}, in which case it might be possible, but I'm not sure about that.
I can definitely add the info at the top of the table definition as a way of seeing it without having to count by hand, but that won't allow you to embed it in other articles.
I guess a parser function would be possible, e.g. {{#CountRows: Table:Companies}}. I haven't investigated them yet, but if that's the kind of thing that will do what you're after then I can look into it. --HappyDog 00:12, 27 March 2007 (BST)
Thanks. It isn't necessary for life, but if it will be not especially complicate work, it will be nice and usefull tool. --195.16.88.26 00:55, 30 March 2007 (BST)
OK - I've done a quick fix that gives the info on the 'data' tab when you're in the table namespace. I'll have a think about other ways of getting this information (suggestions welcome) but you can at least see it easily now. WikiDB.php is the only file that changed for this fix (though there may be other fixes elsewhere, depending how old your copy is). --HappyDog 17:26, 3 April 2007 (BST)
Thank you. Now I know, what big it is and what I can do with it. It's no good way for visitors, but for admin is good. --Neptus 16:04, 21 April 2007 (BST)

Table Formatting

Hi. I love this use for the Wiki, but I've had one frustration - getting formatted tables. To that end, I have made a modification for my own use, and am passing it on. Feel free to include, ignore, whatever. I just thought it might be useful. I have added two internal tags to the repeat command so that there can be footers and headers added to the output produced from the repeat tag.

The trouble was produced because something like:

{|
<repeat table="foo">
|{{{colA}}}
|{{{colB}}}
|-
</repeat>
|}

gives totally useless results, and after looking at the MediaWiki Parser.php file, it turns out that unless that is rewritten, it always will - the table parsing is all one-pass, and doesn't preserve state while in an extension.

So, I have turned that into:

{|
<repeat table="foo">
<header>
{|
</header>
|{{{colA}}}
|{{{colB}}}
|-
<footer>
|}
</footer>
</repeat>
|}

Which now gives the right results. Of course, this requires the following code I added (I know it is rudimentary, but it works...)

Change to WikiDB.php

                $Data = $Table->PerformQuery($Where, $Sort);
		if (trim($Input) == "")
			$Output = $Table->FormatTableData($Data);
		else 
		{
			$outputDef = new WikiDB_OutputFormatDef($Input);
			
			$Output = "";
			$Data = $Table->NormaliseData($Data);
//			print_r($Data);

			if($outputDef->hasHeader())
			{
				$Output .= $outputDef->getHeader() . "\n";
			}
			
			$RowFormat = $outputDef->getRow();
			foreach ($Data as $Row)
			{
				$Output .= pWikiDB_ExpandVariables($RowFormat, $Row);
			}
			
			if($outputDef->hasFooter())
			{
				$Output .= $outputDef->getFooter() . "\n";
			}
//			print($Output);
		}

Also, you must include the following file:

classWikiDB_OutputFormatDef.php

<?php

class WikiDB_OutputFormatDef 
{
	var $sHeader;
	var $sFooter;
	var $sRow;
	
	function WikiDB_OutputFormatDef($contents)
	{
		$this->sHeader = $this->stripTagItem($contents, 'header');
		$this->sFooter = $this->stripTagItem($contents, 'footer');
		$this->sRow		= $contents;
	}
	
	function hasHeader(){ return strlen($this->sHeader) > 0; }
	function hasFooter(){ return strlen($this->sFooter) > 0; }
	
	function getHeader(){ return $this->sHeader; }
	function getFooter(){ return $this->sFooter; }
	function getRow()   { return $this->sRow; }
	
	function stripTagItem(&$text, $tag)
	{
		$tagLen = strlen($tag);
		
		$start = stripos($text, '<' . $tag . '>');
		if($start == FALSE)
		{
			return '';
		}
		
		$end = stripos($text, '</' . $tag . '>');
		if($end == FALSE)
		{
			return '';
		}
		
		$tagTextStart  = $start+$tagLen+2;
		$tagTextLength = $end - $tagTextStart - 1;
		$tagTextEnd    = $end + $tagLen + 3;
		
		$tagContents = substr($text, $tagTextStart, $tagTextLength);
		//strip out the tag
		$text = substr($text, 0, $start) . substr($text, $tagTextEnd);
		
		return $tagContents;
	}
	
}  //END: Class WikiDB_OutputFormatDef

?>

Thanks!

-Jacob 15:50, 8 April 2007 (BST)

Slight Modifications

I integrated your code into my working wiki, because I too saw the uselessness of the table formatting with the repeat tag. However, I made a couple other changes:

  • the table begin and end before and after the repeat are not needed, correct? (the header and footer tags print them)
  • It seems like the table code was inserting an extra line break in the raw input, which was causing a "br" tag in the output, which was skewing the last column of each table row. To change this, I added the last line here:
if (trim($Input) == "")
  $Output = $Table->FormatTableData($Data);
else {
  $Input = trim($Input); // get rid of extra line breaks
  • However, the trim broke the tag processing in the OutputFormatDef file. On the lines "if $start == FALSE" and "if $end == false", I changed the "==" to "===" to be a more correct equality test, since after the trim those tags for me were in position 0 (== FALSE, but not ===FALSE).
  • Finally, "stripos" only works on PHP5, so I changed the 2 lines that use strpos as follows:
OLD: stripos($text, '<' . $tag . '>'));
NEW: strpos($text, strtolower('<' . $tag . '>'));

Thanks for your help and work on this. It's a cool extension.

-- Joe Clark (a guest, joeclark AT joeclarkia DOT net)

Parser Functions & Other Extensions

Due to the way that the WikiDB_Parse function was written, using a local parser it won't call into any extensions that have registered with the global parser (e.g. Parser Functions). However, I have replaced the Parse call with the following, and it seems to be working on my version 1.9.3.

function WikiDB_Parse( $Input, $Parser, $LineStart = true ) 
{
                        $title = $Parser->mTitle;
                        $options = $Parser->mOptions;
                        $output = $Parser->parse( $Input, $title, $options, true, false );
                        return $output->getText();
}

-Jacob 16:49, 14 April 2007 (BST)

Looking at my code, a (slightly different) version of this function already exists. As you don't give details about what code was there before your changes I'm not sure whether this is something I've already fixed, or whether you are pointing out something a bit more subtle. If I don't hear anything else, I'll assume it's fixed, but if there are still problems then please let me know! --HappyDog 12:22, 19 February 2008 (GMT)

Using the <repeat> tag to display filtered/sorted Tables

I have a suggestion to efficiently display tables using the <repeat> tag when a table has a very large number of fields that a user may want to only see a small portion of the fields.

Expand the parameters of the <repeat> tag such that somehow a user can specify which fields to display in the rendered wiki page. Perhaps this need be only one additional parameter that contains a comma separated list of field names to display whilst not displaying all other fields in the specified table.

Example:

Suppose a table called "Stock Investments" contains the following fields: "stock_symbol", "name", "ceo_name", "address", "main_phone", "annual_revenue", "website", "current_stock_price", "my_shares_count".

There are 9 fields in this table such that when the current <repeat> tag was used to display a sorted and/or filtered table of this table, the rendered table in the browser may proceed beyond the right edge of the screen, so perhaps the <repeat> tag could have a new parameter, filter_columns, that is used as such:

<repeat table="Stock Investments" sort="stock_symbol" filter_columns="stock_symbol,name,current_stock_price,my_shares_count,annual_revenue"></repeat>

And this usage of the <repeat> tag would yield a table such as the following with ONLY the columns specified in the filter_columns parameter AND in the order specified.

Stock Investments
stock_symbol name current_stock_price my_shares_count annual_revenue
AMD AMD, Inc 14.30 100 5.649B
KEI Keithley Instruments, Inc 20 12.55 0.155B
MSFT Microsoft, Inc 29.47 25 44.282B

This feature would be very useful and I recommend including support for this feature in WikiDB.

Thanks! -- Mdrayman 22:02, 30 June 2007 (BST)

There are a lot of syntax issues with the repeat tag at present, so I need to get those resolved first. It may be that after the fix-up this kind of thing will be sufficiently simple not to warrant a separate syntax, however if this is not the case then it is a good suggestion, and one I will revisit when the other issues are fixed. --HappyDog 13:35, 4 July 2007 (BST)

Versioning of WikiDB Extension Files

  • Since this Extension is in continual development now toward v1 and will continue in development afterwards, I'd like to suggest a small change to ALL of your WikiDB extension files to contain a Version number so that all of us using the extension can check back here and quickly determine if there have been any updates to the files. A version number can be attached to the $wgExtensionCredits array in each of your php files as shown below for another extension I have installed on my wiki.
$wgExtensionCredits['parserhook'][] = array(
'name' => 'CreateBox',
'url' => 'http://www.mediawiki.org/wiki/Extension:CreateBox',
'description' => 'Specialized Inputbox for page creation',
'author' => 'Ross McClure',
'version' => '1.5'
);
  • Also, what would be even more helpful would be to maintain current version numbers on all files on this page so that all of us using this extension won't have to look in the file sources themselves for the version numbers.
  • Further, since you're working toward v1 right now and common version numbering starts with 0.x.y for pre-v1 release versioning, you could start a version number for your files at 0.1.0 or something similar.
  • Finally, would you consider opening up the development of this extension to the open-source community to allow the inclusion of other code developers/designers/contributors to help develop this extension?

Thanks! -- Mdrayman 04:31, 1 July 2007 (BST)

The problem is that when I work on it, changes go live straight away which means that (a) sometimes the code is broken (hence my big warning) and (b) that it is too much work to do a version change on each update. I store the code in a local Subversion repository, so it would be fairly easy to add the SVN revision to the top of each file - this will probably do the trick. There might be some way of outputting this in some central place as well, but I'm not sure. I will have a think about it... Of course the real solution is to use a separate local testing server (which is what this was, originally, but it has since become the 'public face'), which is something else I might do when I have a bit of time to set one up.
Re: opening up the code - I am happy for people to make suggestions and to send me feature implementations or bug-fixes. I am capable of handling patch files, complete source file with modifications or even just informal code snippets, and of incorporating the changes back into the main branch, and have already done so on a few occasions. However for the time being I want to keep everything centralised here. Once we get a bit nearer v1, and the spec. is more clearly defined (particularly re: syntax issues) I will look into adding it to the MW SVN code repository. --HappyDog 13:48, 4 July 2007 (BST)
I have added SVN revision tags to the files, which contain the most recent revision no. of the files. I have also added a new feature to my WikiDebug extension to display this info in-wiki, and have added this functionality to the files page. I hope this solves the problem for you (feedback is always welcome!) --HappyDog 00:50, 13 July 2007 (BST)
The latest version (which is not yet posted here) has also added code to display the version number on Special:Version. Check that page, and if a version no. is being displayed then I have updated the site with this new feature.
Regarding the overall versioning of the software, I have decided to adopt a scheme whereby I don't have to remember to update version numbers every time I update the files (as the update often happens automatically). The revision number is automatically incremented with every change, plus there is a version number that I manually bump whenever there is a schema or stored-date-format change. This will be available in the next version, when it is made live. --HappyDog 12:22, 19 February 2008 (GMT)

Create Forms to fill in Table data

I'd like to suggest to everyone watching/discussing/contributing to WikiDB that we investigate a way (or create our own way) to create forms inside wiki pages that will actually fill in the fields of a <data> tag statement so that data in the tables can be easily entered without manual editing of the wiki page that contains the record entry via the <data> tag.

I've come across several "forms" style extensions for MediaWik such as Simple Forms, FormTools, Semantic Forms, however, I haven't found one that will work, either because it still eludes me or I don't understand how to make use of one or more of the extensions I listed here.

Anybody have any ideas about how to create forms to fill fields in a <data> tag statement inside a wiki article?

Javascript, perhaps ???

An expansion of WikiDB functionality ???

-- Mdrayman 04:48, 3 July 2007 (BST)

I wasn't considering this for v1, but I was planning it for a later version. It should be relatively simple to use a form to create data, though I am not sure how easy it would be to edit data in this fashion. I'm sure it will be possible, but when editing it will be necessary to gracefully handle all sorts of cases where the existing data doesn't validate against the existing table def, as well as being able to write the data back to the page without screwing things up... Any suggestions, or even code, is welcome. --HappyDog 13:51, 4 July 2007 (BST)

Hyperlink reference to Table:Name when editing page

When editing a page in the main namespace that includes some <data> tags, thought it would be a nice idea to also include a hyperlink at the bottom to the corresponding Table: page (if it exists!). A bit like how the Template: links are provided. - Cocjh1 23:43, 27 November 2007 (GMT)

Adding table properties within repeat tag

Could you please provide a patch which enables extensions to <repeat> tag for adding custom attributes for <table> generated by default? This can be the alternative to missing table formating within <repeat>. Especially, I need "style" and "class" attributes to be added to default table output so I can format this table as required and apply the sortable property.

Hi - The table formatting code is in a bit of a mess at the moment (as is quite a lot of the code...!) and I am in the process of tidying it all up at the moment. I'm quite a long way through this process, but haven't touched on the formatting yet. When I do I'll make it a bit more flexible. All of the major elements that the extension outputs will have a class defined for them, so you can style these as you need. If you need the ability to add your own custom classes as well then I don't see a problem with adding that feature. --HappyDog 12:05, 5 February 2008 (GMT)

Some display suggestions

HappyDog, this is a great extension. I have a couple of suggestions that may make it even more useful or adoptable:

  • Accessing a Table:... page would show the Data by default, vice the table schema. This would allow people to avoid using the repeat construct and just interlink to the table for a particular data item directly. Or the "article" view for a Table page could be what is currently the data view.
  • The Table data display could use the class wikitable sortable and then you would be less reliant again on the repeat tag for filtering, sorting. Furthermore, some AJAX wiz could implement the filtering on the client side for the Table data display.
  • Enable table data column headers to have wiki markup, so you could have a definition (or at least alt- text) for each column header so users could read more about a particular field. This could be specified in the field specification on the schema page.

Please let me know if you have any questions about these goofy requests! :) 13:08, 30 September 2008 (BST)