Talk:WikiDB

From TestWiki
Revision as of 23:43, 27 November 2007 by Cocjh1 (Talk | contribs) (Hyperlink reference to Table:Name when editing page)

Jump to: navigation, search

External queries

This extension is really cool. Do you have any plans concerning external (read-only) queries to the database? --Martin 23:51, 10 July 2006 (BST)

Can you please expand on this - I'm not sure what you mean. --HappyDog 23:55, 10 July 2006 (BST)
Sure. Suppose I have wiki site which is primarily a user-editable database. Each wiki page may provide several database records (let's say information on sequences of proteins of a given family). There is some text which gives some information about the link between these database entries, that's the traditional wiki part. This cannot be put into the database because it's changing often, it may become large, and it doesn't have a meaningful format--the only field name which would be appropriate would be "comments". But the database records are essential for let's say the community of molecular biologists, who use client software to query and analyze the protein from the database. There may be wikis which want to use the database records as illustration, or maybe group the database records differently and comment on them. The idea is to not duplicate the database, but use feeds which would be updated and propagated from the source. My point of view comes from what I did for inserting bibliographic records (PMIDs from Medline and ISBNs from isbndb.org), in wikis: see Biblio extension. Someone also pointed out the Wikicite project. I don't know where these folks are going exactly, but the idea is to build a database of bibliographic references. Have a look. --Martin 00:19, 11 July 2006 (BST)
I haven't really given this any thought yet, and I will take a look at the links shortly, but on the surface what you say sounds quite plausible. How I would implement this is to use a new action argument, e.g. $action=rawdata (or an existing action if a suitable one exists - raw?). So if you entered e.g. http://some.domain.org/w/index.php?title=Table:Companies&action=rawdata, you would get the data (and schema?) from the Companies table in a standard format, probably XML. Does this sound like it would do the trick? (Bear in mind that if the table definition is altered, the XML output will change) --HappyDog 00:58, 11 July 2006 (BST)
I was thinking of something like an SQL SELECT statement. More specifically, the only features that I think would be useful would be:
  • download the whole database for local use as you just mentioned.
  • select one or several records by giving one column name and passing a list of strings. All records for which one of those strings matches is selected.
I don't know if you can do this simply using MediaWiki. Basically the URL I would pass would be like http://some.domain.org/w/index.php?title=Table:Companies&action=rawdata&select_id=12,34,567 where id is a column name and 12, 34 and 567 are acceptable id values. Maybe it is easier to just write an independent PHP script which queries the database according to the CGI arguments which are found. --Martin 01:40, 11 July 2006 (BST)
I think that level of detail is outside the scope of v1, however you're right, it would be fairly trivial to write a PHP script to parse and manipulate the raw data dump, particularly if it is in a standard format such as XML (or, it just occurs to me, SQL - I'm sure that would be possible, and then you could simply import it straight into a traditional database... hmmm... interesting thought!) --HappyDog 02:45, 11 July 2006 (BST)

Fetch and use elements of a table

Hi!

As you know, I expect this extension for a long time. Beside his earlier stage, it is a very important step, if we think to use MediaWiki seriously, in my point of view.

Well, my question is about fetch elements in the DB and use them in a infobox or elsewhere in a page. I think the <repeat> tag may be used for that, but I think this tag automatically shows a table with the data fetched.

What I need to do is something like:

 This are the companies at the DB:
 <repeat table="Companies3" criteria="Name!=Microsoft" sort="Logo desc">
   {{name}} is founded at {{founded}} and have a revenue of {{revenue}}
 </repeat>

which produces the next output:

Intel is founded at USA and have a revenue of #N/A     
Apple is founded at April 2005 and have a revenue of $7.42

Maybe we can even use the <repeat> tag with a additional template option, as in

<repeat table="Companies3" criteria="Name!=Microsoft" sort="Logo desc" template="Infobox company">
</repeat>


Best regards and again, thanks for your effort

Ivo Almeida

Hi Ivo - the first example you give is exactly how I plan the repeat tag to work! I have not coded it yet, but basically the contents of the repeat tag will be repeated for each row, with variable substitution working the same way as for templates (I may even harness the template code to do this, if that is possible...). If you want to send me your e-mail address I will let you know when this change is in place. Alternatively, just check back here from time to time. Thanks for your interest. --HappyDog 17:20, 20 September 2006 (BST)
I added this functionality a little while ago, but forgot to update this page. There are still a few bugs which need to be ironed out, but the general case works. --HappyDog 19:10, 1 December 2006 (GMT)

WikiDB hate math?

It's imposible to use math and WikiDB on one page. My wiki have a lot of math formulas. I try build special dictionary using WikiDB. If I put on page list of WikiDB records, then any math no works on this page. :(( I use MimeTeX as cgi. --195.16.88.26 16:03, 22 January 2007 (GMT)

Hi - Can you give me an example of the markup that you use so that I can investigate further. Thanks --HappyDog 16:54, 22 January 2007 (GMT)

Thanks for interest. WikiDB is very pleasant extension. I contrive now, how to do nice search for it and it will be beatifull then.

Test page with WikiDB (You can destroy this page, it's only test for you): http://www.astro.hg.pl/wiki/algorytmy/index.php?title=Test_WikiDB%26Math

The same original page without WikiDB: http://www.astro.hg.pl/wiki/algorytmy/index.php?title=Ascendent_i_Descendent

Link for separate page with DB list is no problem here. I only sygnalize you conflict.


Definition of DB: http://www.astro.hg.pl/wiki/algorytmy/index.php?title=Slownik:Slowka

Definitions of listed records: http://www.astro.hg.pl/wiki/algorytmy/index.php?title=S%C5%82ownik:Ascendent

It can be even :<math>a</math> and listing of one record of DB. Effect is identical like definition of data and listing on the same page: error type ?UNIQ35cb8610deb4633-math-4952b6c343f9c54300000002-QINU instead of formula.--195.16.88.26 21:04, 22 January 2007 (GMT)

Thanks for the examples - I will look into it and post my findings here. --HappyDog 21:29, 22 January 2007 (GMT)
OK - I have managed to get textvc installed on my local copy, so I can enable math tags (for a proper test). I tried it with MW1.6.8 (as installed on your server) and I can replicate the problem. Interestingly, it is not a problem in MW1.5.6, which is what I have been working with up to now. I will look further and see what's causing the problem... --HappyDog 02:07, 23 January 2007 (GMT)
Fixed it! Download the latest version of WikiDB.php and you should be fine.
The problem was the way that I was accessing the parser - it caused it to lose the information about tokens that had already been stripped (in this case the math tag). Using a new Parser instance avoids this. Let me know if you have any other problems or questions! --HappyDog 02:59, 23 January 2007 (GMT)
Oh, yes! Now is excellent! Thanks. :)) --Neptus (195.16.88.26) 09:02, 2 February 2007 (GMT)

WikiDB is too slow

If number of base records grows, it is great problem. Server generates timeout and pages with queries no works. I have about 900 simple record definitions now.

Complet list works:

http://www.astro.hg.pl/wiki/algorytmy/index.php?title=S%C5%82ownik:Wszystkie_has%C5%82a

When I use selection, i.e.:

  <repeat table="Slowka" criteria="kod=en" sort="tekst"># {{{tekst}}} = {{{definicja}}}
</repeat>

- it no works because of very long time needing for this operation, considerably higher above server time limits. http://www.astro.hg.pl/wiki/algorytmy/index.php?title=English

Server generates error(500). It means in short translation: script needs too much server processor time or server memory or in .htaccess file contains any illegal directiv. --Neptus 17:21, 11 February 2007 (GMT)

As it says at the top of the page, there is loads of optimisation that needs to be done to this to make it anywhere near usable for large data sets. I won't be working on the optimisation of this code until the core functionality is complete (although anyone else is free to try or to offer suggestions for optimisation). In the meantime, all I can suggest is to increase max_execution_time for PHP (see [1] for one method). This will fix the time-out issue but won't increase the rendering time - i.e. it will still take 30+ seconds, but you will get to view the result (eventually...). --HappyDog 22:16, 11 February 2007 (GMT)
It's only virtual server, not my own PHP server. :( In that case, I must rewrite it to other format... Thanks. --Neptus 22:50, 11 February 2007 (GMT)
All is correct. It was malicious server error - only this one account from a lot of thousends on server works bad. Sorry. WikiDB is really slow, but it works in reasonable time. All my pages are viewed in a few seconds now. I still like and use this extension. :) --Neptus 21:09, 13 February 2007 (GMT)

WikiDB does not work with mediawiki 1.9.1 ?

After installed the extension to mediawiki-1.9.1, I tested and it seems that WikiDB is able to create a table and data can be recorded in the database. However, the data cannot be showing up on a wiki page. Neither using "table" or using "repeat" tag.

For example, write the following code to a page called Sandbox,

<data table="Companies" template="Infobox Company">
name=Microsoft 
founded=1492 
revenue=$8 
</data>

the browser address bar shows

http://mysiteurl.com/wiki/index.php?title=Sandbox&action=submit

and the sanbox become an empty blank page

Checking the undefined table spacial page, I do see the Table:Companies and check the data of the table, I got the following:

Defined Fields Undefined Fields
Source Article name founded revenue
Sandbox Microsoft 1492 $8

After that, the sandbox page is not accessable any more. Or it shows as a blank page, I cannot edit/delete it

If I define the table first and then use "table" tag to save some data, the data will be saved to the table, but using "repeat" to retrive the data, the data cannot be shown on a page, again, the page shows blank.

Is wikiDB tested in mediawiki 1.9.1?

No - I haven't tested WikiDB with later versions of MW yet, but I will do so as soon as I have a bit of time and let you know how I get on right here. --HappyDog 23:15, 14 March 2007 (GMT)
OK - I've fixed this bug. As far as I know WikiDB now works on MW 1.9.1. You will need to download the latest version of WikiDB.php, but the rest of the files remain unchanged. Please let me know if you encounter any more bugs.
Here's the technical bit for those who are interested: The Parser::parse() function takes a 'clearstate' argument, which if true resets the state of the parser, and if false leaves it as it is. Because I was calling a new instance of the parser with this flag set to false, it was not being initialised properly, and so a PHP error was being raised. Presumably your system is set up to not display errors, hence the blank page. This has clearly changed from earlier versions, whether deliberately or not, which is why there was no problem on this wiki (currently running 1.5.8). Anyway, I have now set the clearstate argument to true and everything seems to work fine on both wikis (fingers crossed!) :-) --HappyDog 10:38, 29 March 2007 (BST)
Thanks, Sir! This fix the problem and it works on Mediawiki 1.9.3 as well! :)

And with mediawiki 1.10.0? Any successful test?

Hi, I'm not sure who asked this question, but I have successfully tested some of the basic functionality on my MediaWiki 1.10.0 installation. The details are here on the Tables Talk page under Creating Tables for WikiDB Extension.

Let me know if you have any questions.

--Mdrayman 05:34, 25 June 2007 (BST)

simple way to do WikiDB faster

Add second single index row_id to table wikidb_fielddata. My WikiDB is like a rocket now! --Neptus 15:06, 21 February 2007 (GMT)

Hi Neptus. Sorry about the slow response - I've been away. I've checked the table definitions and this index is already there (3rd row in the index table). Perhaps you forgot to add it when you setup your tables originally (I'm pretty sure it's been there since the start). Or maybe I've misunderstood what you are saying? --HappyDog 23:15, 14 March 2007 (GMT)
Ooops.. You are right! Then I'm a blondie... I've put it all to one index... Sorry, then your base isn't slow. It is very fast base. --Neptus 02:11, 17 March 2007 (GMT)

Count of records

Is it possible to get number of record definitions without listing all? --Neptus 21:34, 14 March 2007 (GMT)

Can you please elaborate. Do you mean via a tag (so it can be displayed in a page) or on the table definition. I'm sure it would be fairly easy to add in either case - please tell me what you would like to see and where, and I'll see what I can do. --HappyDog 23:15, 14 March 2007 (GMT)
I have 4000 definitions in my WikiDB dictonary now and it grows all the time. I can count it only using tag <repeat table...># ... </repeat> now. This listing is long.
It will be nice to have only one aktual number of positions declared in WikiDB on page (or a number of selected records), visual effect like a number of pages {{NUMBEROFARTICLES}} in wiki - simple base counter: how many records is inside. Nice gadget for information and for a boast. Then it will be a tag, I think. --Neptus 01:40, 17 March 2007 (GMT)
The problem is that it would be a single number for the whole wiki. If I added a {{NUMBEROFROWS}} magic word it would give the no. of rows in all tables in the wiki, combined. It might be possible to somehow supply an argument to a magic word (e.g. {{NUMBEROFROWS|Table:Companies}}, in which case it might be possible, but I'm not sure about that.
I can definitely add the info at the top of the table definition as a way of seeing it without having to count by hand, but that won't allow you to embed it in other articles.
I guess a parser function would be possible, e.g. {{#CountRows: Table:Companies}}. I haven't investigated them yet, but if that's the kind of thing that will do what you're after then I can look into it. --HappyDog 00:12, 27 March 2007 (BST)
Thanks. It isn't necessary for life, but if it will be not especially complicate work, it will be nice and usefull tool. --195.16.88.26 00:55, 30 March 2007 (BST)
OK - I've done a quick fix that gives the info on the 'data' tab when you're in the table namespace. I'll have a think about other ways of getting this information (suggestions welcome) but you can at least see it easily now. WikiDB.php is the only file that changed for this fix (though there may be other fixes elsewhere, depending how old your copy is). --HappyDog 17:26, 3 April 2007 (BST)
Thank you. Now I know, what big it is and what I can do with it. It's no good way for visitors, but for admin is good. --Neptus 16:04, 21 April 2007 (BST)

Table Formatting

Hi. I love this use for the Wiki, but I've had one frustration - getting formatted tables. To that end, I have made a modification for my own use, and am passing it on. Feel free to include, ignore, whatever. I just thought it might be useful. I have added two internal tags to the repeat command so that there can be footers and headers added to the output produced from the repeat tag.

The trouble was produced because something like:

{|
<repeat table="foo">
|{{{colA}}}
|{{{colB}}}
|-
</repeat>
|}

gives totally useless results, and after looking at the MediaWiki Parser.php file, it turns out that unless that is rewritten, it always will - the table parsing is all one-pass, and doesn't preserve state while in an extension.

So, I have turned that into:

{|
<repeat table="foo">
<header>
{|
</header>
|{{{colA}}}
|{{{colB}}}
|-
<footer>
|}
</footer>
</repeat>
|}

Which now gives the right results. Of course, this requires the following code I added (I know it is rudimentary, but it works...)

Change to WikiDB.php

                $Data = $Table->PerformQuery($Where, $Sort);
		if (trim($Input) == "")
			$Output = $Table->FormatTableData($Data);
		else 
		{
			$outputDef = new WikiDB_OutputFormatDef($Input);
			
			$Output = "";
			$Data = $Table->NormaliseData($Data);
//			print_r($Data);

			if($outputDef->hasHeader())
			{
				$Output .= $outputDef->getHeader() . "\n";
			}
			
			$RowFormat = $outputDef->getRow();
			foreach ($Data as $Row)
			{
				$Output .= pWikiDB_ExpandVariables($RowFormat, $Row);
			}
			
			if($outputDef->hasFooter())
			{
				$Output .= $outputDef->getFooter() . "\n";
			}
//			print($Output);
		}

Also, you must include the following file:

classWikiDB_OutputFormatDef.php

<?php

class WikiDB_OutputFormatDef 
{
	var $sHeader;
	var $sFooter;
	var $sRow;
	
	function WikiDB_OutputFormatDef($contents)
	{
		$this->sHeader = $this->stripTagItem($contents, 'header');
		$this->sFooter = $this->stripTagItem($contents, 'footer');
		$this->sRow		= $contents;
	}
	
	function hasHeader(){ return strlen($this->sHeader) > 0; }
	function hasFooter(){ return strlen($this->sFooter) > 0; }
	
	function getHeader(){ return $this->sHeader; }
	function getFooter(){ return $this->sFooter; }
	function getRow()   { return $this->sRow; }
	
	function stripTagItem(&$text, $tag)
	{
		$tagLen = strlen($tag);
		
		$start = stripos($text, '<' . $tag . '>');
		if($start == FALSE)
		{
			return '';
		}
		
		$end = stripos($text, '</' . $tag . '>');
		if($end == FALSE)
		{
			return '';
		}
		
		$tagTextStart  = $start+$tagLen+2;
		$tagTextLength = $end - $tagTextStart - 1;
		$tagTextEnd    = $end + $tagLen + 3;
		
		$tagContents = substr($text, $tagTextStart, $tagTextLength);
		//strip out the tag
		$text = substr($text, 0, $start) . substr($text, $tagTextEnd);
		
		return $tagContents;
	}
	
}  //END: Class WikiDB_OutputFormatDef

?>

Thanks!

-Jacob 15:50, 8 April 2007 (BST)

Slight Modifications

I integrated your code into my working wiki, because I too saw the uselessness of the table formatting with the repeat tag. However, I made a couple other changes:

  • the table begin and end before and after the repeat are not needed, correct? (the header and footer tags print them)
  • It seems like the table code was inserting an extra line break in the raw input, which was causing a "br" tag in the output, which was skewing the last column of each table row. To change this, I added the last line here:
if (trim($Input) == "")
  $Output = $Table->FormatTableData($Data);
else {
  $Input = trim($Input); // get rid of extra line breaks
  • However, the trim broke the tag processing in the OutputFormatDef file. On the lines "if $start == FALSE" and "if $end == false", I changed the "==" to "===" to be a more correct equality test, since after the trim those tags for me were in position 0 (== FALSE, but not ===FALSE).
  • Finally, "stripos" only works on PHP5, so I changed the 2 lines that use strpos as follows:
OLD: stripos($text, '<' . $tag . '>'));
NEW: strpos($text, strtolower('<' . $tag . '>'));

Thanks for your help and work on this. It's a cool extension.

-- Joe Clark (a guest, joeclark AT joeclarkia DOT net)

Parser Functions & Other Extensions

Due to the way that the WikiDB_Parse function was written, using a local parser it won't call into any extensions that have registered with the global parser (e.g. Parser Functions). However, I have replaced the Parse call with the following, and it seems to be working on my version 1.9.3.

function WikiDB_Parse( $Input, $Parser, $LineStart = true ) 
{
                        $title = $Parser->mTitle;
                        $options = $Parser->mOptions;
                        $output = $Parser->parse( $Input, $title, $options, true, false );
                        return $output->getText();
}

-Jacob 16:49, 14 April 2007 (BST)

Using the <repeat> tag to display filtered/sorted Tables

I have a suggestion to efficiently display tables using the <repeat> tag when a table has a very large number of fields that a user may want to only see a small portion of the fields.

Expand the parameters of the <repeat> tag such that somehow a user can specify which fields to display in the rendered wiki page. Perhaps this need be only one additional parameter that contains a comma separated list of field names to display whilst not displaying all other fields in the specified table.

Example:

Suppose a table called "Stock Investments" contains the following fields: "stock_symbol", "name", "ceo_name", "address", "main_phone", "annual_revenue", "website", "current_stock_price", "my_shares_count".

There are 9 fields in this table such that when the current <repeat> tag was used to display a sorted and/or filtered table of this table, the rendered table in the browser may proceed beyond the right edge of the screen, so perhaps the <repeat> tag could have a new parameter, filter_columns, that is used as such:

<repeat table="Stock Investments" sort="stock_symbol" filter_columns="stock_symbol,name,current_stock_price,my_shares_count,annual_revenue"></repeat>

And this usage of the <repeat> tag would yield a table such as the following with ONLY the columns specified in the filter_columns parameter AND in the order specified.

Stock Investments
stock_symbol name current_stock_price my_shares_count annual_revenue
AMD AMD, Inc 14.30 100 5.649B
KEI Keithley Instruments, Inc 20 12.55 0.155B
MSFT Microsoft, Inc 29.47 25 44.282B

This feature would be very useful and I recommend including support for this feature in WikiDB.

Thanks! -- Mdrayman 22:02, 30 June 2007 (BST)

There are a lot of syntax issues with the repeat tag at present, so I need to get those resolved first. It may be that after the fix-up this kind of thing will be sufficiently simple not to warrant a separate syntax, however if this is not the case then it is a good suggestion, and one I will revisit when the other issues are fixed. --HappyDog 13:35, 4 July 2007 (BST)

Versioning of WikiDB Extension Files

  • Since this Extension is in continual development now toward v1 and will continue in development afterwards, I'd like to suggest a small change to ALL of your WikiDB extension files to contain a Version number so that all of us using the extension can check back here and quickly determine if there have been any updates to the files. A version number can be attached to the $wgExtensionCredits array in each of your php files as shown below for another extension I have installed on my wiki.
$wgExtensionCredits['parserhook'][] = array(
'name' => 'CreateBox',
'url' => 'http://www.mediawiki.org/wiki/Extension:CreateBox',
'description' => 'Specialized Inputbox for page creation',
'author' => 'Ross McClure',
'version' => '1.5'
);
  • Also, what would be even more helpful would be to maintain current version numbers on all files on this page so that all of us using this extension won't have to look in the file sources themselves for the version numbers.
  • Further, since you're working toward v1 right now and common version numbering starts with 0.x.y for pre-v1 release versioning, you could start a version number for your files at 0.1.0 or something similar.
  • Finally, would you consider opening up the development of this extension to the open-source community to allow the inclusion of other code developers/designers/contributors to help develop this extension?

Thanks! -- Mdrayman 04:31, 1 July 2007 (BST)

The problem is that when I work on it, changes go live straight away which means that (a) sometimes the code is broken (hence my big warning) and (b) that it is too much work to do a version change on each update. I store the code in a local Subversion repository, so it would be fairly easy to add the SVN revision to the top of each file - this will probably do the trick. There might be some way of outputting this in some central place as well, but I'm not sure. I will have a think about it... Of course the real solution is to use a separate local testing server (which is what this was, originally, but it has since become the 'public face'), which is something else I might do when I have a bit of time to set one up.
Re: opening up the code - I am happy for people to make suggestions and to send me feature implementations or bug-fixes. I am capable of handling patch files, complete source file with modifications or even just informal code snippets, and of incorporating the changes back into the main branch, and have already done so on a few occasions. However for the time being I want to keep everything centralised here. Once we get a bit nearer v1, and the spec. is more clearly defined (particularly re: syntax issues) I will look into adding it to the MW SVN code repository. --HappyDog 13:48, 4 July 2007 (BST)
I have added SVN revision tags to the files, which contain the most recent revision no. of the files. I have also added a new feature to my WikiDebug extension to display this info in-wiki, and have added this functionality to the files page. I hope this solves the problem for you (feedback is always welcome!) --HappyDog 00:50, 13 July 2007 (BST)

Create Forms to fill in Table data

I'd like to suggest to everyone watching/discussing/contributing to WikiDB that we investigate a way (or create our own way) to create forms inside wiki pages that will actually fill in the fields of a <data> tag statement so that data in the tables can be easily entered without manual editing of the wiki page that contains the record entry via the <data> tag.

I've come across several "forms" style extensions for MediaWik such as Simple Forms, FormTools, Semantic Forms, however, I haven't found one that will work, either because it still eludes me or I don't understand how to make use of one or more of the extensions I listed here.

Anybody have any ideas about how to create forms to fill fields in a <data> tag statement inside a wiki article?

Javascript, perhaps ???

An expansion of WikiDB functionality ???

-- Mdrayman 04:48, 3 July 2007 (BST)

I wasn't considering this for v1, but I was planning it for a later version. It should be relatively simple to use a form to create data, though I am not sure how easy it would be to edit data in this fashion. I'm sure it will be possible, but when editing it will be necessary to gracefully handle all sorts of cases where the existing data doesn't validate against the existing table def, as well as being able to write the data back to the page without screwing things up... Any suggestions, or even code, is welcome. --HappyDog 13:51, 4 July 2007 (BST)

Current Status?

It looks like there hasn't been a lot of development here since July or so. What's the current status? This seems like a cool extension, so I hope development will continue. - Joeclark 05:53, 21 November 2007 (GMT)

Development will definitely continue, but work and a few other more urgent projects have taken over a little at the moment. I will be revisiting this as soon as I have a bit of free time. In the meantime if you have a specific issue that needs addressing urgently then let me know and I'll try and squeeze it into my schedule. Thanks for your interest --HappyDog 15:27, 22 November 2007 (GMT)

Hyperlink reference to Table:Name when editing page

When editing a page in the main namespace that includes some <data> tags, thought it would be a nice idea to also include a hyperlink at the bottom to the corresponding Table: page (if it exists!). A bit like how the Template: links are provided. - Cocjh1 23:43, 27 November 2007 (GMT)