Difference between revisions of "Talk:WikiDB"

From TestWiki
Jump to: navigation, search
(WikiDB does not work with mediawiki 1.9.1 ?)
(And with mediawiki 1.10.0? Any successful test?)
Line 162: Line 162:
  
 
== And with mediawiki 1.10.0? Any successful test? ==
 
== And with mediawiki 1.10.0? Any successful test? ==
 +
 +
Hi, I'm not sure who asked this question, but I have successfully tested some of the basic functionality on my MediaWiki 1.10.0 installation.  The details are [[Talk:WikiDB/Tables#Creating Tables for WikiDB extension|here on the Tables Talk page]] under '''Creating Tables for WikiDB Extension'''.
 +
 +
Let me know if you have any questions.
 +
 +
--[[User:Mdrayman|Mdrayman]] 05:34, 25 June 2007 (BST)
  
 
== simple way to do WikiDB faster ==
 
== simple way to do WikiDB faster ==

Revision as of 04:34, 25 June 2007

This extension is really cool. Do you have any plans concerning external (read-only) queries to the database? --Martin 23:51, 10 July 2006 (BST)

Can you please expand on this - I'm not sure what you mean. --HappyDog 23:55, 10 July 2006 (BST)
Sure. Suppose I have wiki site which is primarily a user-editable database. Each wiki page may provide several database records (let's say information on sequences of proteins of a given family). There is some text which gives some information about the link between these database entries, that's the traditional wiki part. This cannot be put into the database because it's changing often, it may become large, and it doesn't have a meaningful format--the only field name which would be appropriate would be "comments". But the database records are essential for let's say the community of molecular biologists, who use client software to query and analyze the protein from the database. There may be wikis which want to use the database records as illustration, or maybe group the database records differently and comment on them. The idea is to not duplicate the database, but use feeds which would be updated and propagated from the source. My point of view comes from what I did for inserting bibliographic records (PMIDs from Medline and ISBNs from isbndb.org), in wikis: see Biblio extension. Someone also pointed out the Wikicite project. I don't know where these folks are going exactly, but the idea is to build a database of bibliographic references. Have a look. --Martin 00:19, 11 July 2006 (BST)
I haven't really given this any thought yet, and I will take a look at the links shortly, but on the surface what you say sounds quite plausible. How I would implement this is to use a new action argument, e.g. $action=rawdata (or an existing action if a suitable one exists - raw?). So if you entered e.g. http://some.domain.org/w/index.php?title=Table:Companies&action=rawdata, you would get the data (and schema?) from the Companies table in a standard format, probably XML. Does this sound like it would do the trick? (Bear in mind that if the table definition is altered, the XML output will change) --HappyDog 00:58, 11 July 2006 (BST)
I was thinking of something like an SQL SELECT statement. More specifically, the only features that I think would be useful would be:
  • download the whole database for local use as you just mentioned.
  • select one or several records by giving one column name and passing a list of strings. All records for which one of those strings matches is selected.
I don't know if you can do this simply using MediaWiki. Basically the URL I would pass would be like http://some.domain.org/w/index.php?title=Table:Companies&action=rawdata&select_id=12,34,567 where id is a column name and 12, 34 and 567 are acceptable id values. Maybe it is easier to just write an independent PHP script which queries the database according to the CGI arguments which are found. --Martin 01:40, 11 July 2006 (BST)
I think that level of detail is outside the scope of v1, however you're right, it would be fairly trivial to write a PHP script to parse and manipulate the raw data dump, particularly if it is in a standard format such as XML (or, it just occurs to me, SQL - I'm sure that would be possible, and then you could simply import it straight into a traditional database... hmmm... interesting thought!) --HappyDog 02:45, 11 July 2006 (BST)

Fetch and use elements of a table

Hi!

As you know, I expect this extension for a long time. Beside his earlier stage, it is a very important step, if we think to use MediaWiki seriously, in my point of view.

Well, my question is about fetch elements in the DB and use them in a infobox or elsewhere in a page. I think the <repeat> tag may be used for that, but I think this tag automatically shows a table with the data fetched.

What I need to do is something like:

 This are the companies at the DB:
 <repeat table="Companies3" criteria="Name!=Microsoft" sort="Logo desc">
   {{name}} is founded at {{founded}} and have a revenue of {{revenue}}
 </repeat>

which produces the next output:

Intel is founded at USA and have a revenue of #N/A     
Apple is founded at April 2005 and have a revenue of $7.42

Maybe we can even use the <repeat> tag with a additional template option, as in

<repeat table="Companies3" criteria="Name!=Microsoft" sort="Logo desc" template="Infobox company">
</repeat>


Best regards and again, thanks for your effort

Ivo Almeida

Hi Ivo - the first example you give is exactly how I plan the repeat tag to work! I have not coded it yet, but basically the contents of the repeat tag will be repeated for each row, with variable substitution working the same way as for templates (I may even harness the template code to do this, if that is possible...). If you want to send me your e-mail address I will let you know when this change is in place. Alternatively, just check back here from time to time. Thanks for your interest. --HappyDog 17:20, 20 September 2006 (BST)
I added this functionality a little while ago, but forgot to update this page. There are still a few bugs which need to be ironed out, but the general case works. --HappyDog 19:10, 1 December 2006 (GMT)

WikiDB hate math?

It's imposible to use math and WikiDB on one page. My wiki have a lot of math formulas. I try build special dictionary using WikiDB. If I put on page list of WikiDB records, then any math no works on this page. :(( I use MimeTeX as cgi. --195.16.88.26 16:03, 22 January 2007 (GMT)

Hi - Can you give me an example of the markup that you use so that I can investigate further. Thanks --HappyDog 16:54, 22 January 2007 (GMT)

Thanks for interest. WikiDB is very pleasant extension. I contrive now, how to do nice search for it and it will be beatifull then.

Test page with WikiDB (You can destroy this page, it's only test for you): http://www.astro.hg.pl/wiki/algorytmy/index.php?title=Test_WikiDB%26Math

The same original page without WikiDB: http://www.astro.hg.pl/wiki/algorytmy/index.php?title=Ascendent_i_Descendent

Link for separate page with DB list is no problem here. I only sygnalize you conflict.


Definition of DB: http://www.astro.hg.pl/wiki/algorytmy/index.php?title=Slownik:Slowka

Definitions of listed records: http://www.astro.hg.pl/wiki/algorytmy/index.php?title=S%C5%82ownik:Ascendent

It can be even :<math>a</math> and listing of one record of DB. Effect is identical like definition of data and listing on the same page: error type ?UNIQ35cb8610deb4633-math-4952b6c343f9c54300000002-QINU instead of formula.--195.16.88.26 21:04, 22 January 2007 (GMT)

Thanks for the examples - I will look into it and post my findings here. --HappyDog 21:29, 22 January 2007 (GMT)
OK - I have managed to get textvc installed on my local copy, so I can enable math tags (for a proper test). I tried it with MW1.6.8 (as installed on your server) and I can replicate the problem. Interestingly, it is not a problem in MW1.5.6, which is what I have been working with up to now. I will look further and see what's causing the problem... --HappyDog 02:07, 23 January 2007 (GMT)
Fixed it! Download the latest version of WikiDB.php and you should be fine.
The problem was the way that I was accessing the parser - it caused it to lose the information about tokens that had already been stripped (in this case the math tag). Using a new Parser instance avoids this. Let me know if you have any other problems or questions! --HappyDog 02:59, 23 January 2007 (GMT)
Oh, yes! Now is excellent! Thanks. :)) --Neptus (195.16.88.26) 09:02, 2 February 2007 (GMT)

WikiDB is too slow

If number of base records grows, it is great problem. Server generates timeout and pages with queries no works. I have about 900 simple record definitions now.

Complet list works:

http://www.astro.hg.pl/wiki/algorytmy/index.php?title=S%C5%82ownik:Wszystkie_has%C5%82a

When I use selection, i.e.:

  <repeat table="Slowka" criteria="kod=en" sort="tekst"># {{{tekst}}} = {{{definicja}}}
</repeat>

- it no works because of very long time needing for this operation, considerably higher above server time limits. http://www.astro.hg.pl/wiki/algorytmy/index.php?title=English

Server generates error(500). It means in short translation: script needs too much server processor time or server memory or in .htaccess file contains any illegal directiv. --Neptus 17:21, 11 February 2007 (GMT)

As it says at the top of the page, there is loads of optimisation that needs to be done to this to make it anywhere near usable for large data sets. I won't be working on the optimisation of this code until the core functionality is complete (although anyone else is free to try or to offer suggestions for optimisation). In the meantime, all I can suggest is to increase max_execution_time for PHP (see [1] for one method). This will fix the time-out issue but won't increase the rendering time - i.e. it will still take 30+ seconds, but you will get to view the result (eventually...). --HappyDog 22:16, 11 February 2007 (GMT)
It's only virtual server, not my own PHP server. :( In that case, I must rewrite it to other format... Thanks. --Neptus 22:50, 11 February 2007 (GMT)
All is correct. It was malicious server error - only this one account from a lot of thousends on server works bad. Sorry. WikiDB is really slow, but it works in reasonable time. All my pages are viewed in a few seconds now. I still like and use this extension. :) --Neptus 21:09, 13 February 2007 (GMT)

WikiDB does not work with mediawiki 1.9.1 ?

After installed the extension to mediawiki-1.9.1, I tested and it seems that WikiDB is able to create a table and data can be recorded in the database. However, the data cannot be showing up on a wiki page. Neither using "table" or using "repeat" tag.

For example, write the following code to a page called Sandbox,

<data table="Companies" template="Infobox Company">
name=Microsoft 
founded=1492 
revenue=$8 
</data>

the browser address bar shows

http://mysiteurl.com/wiki/index.php?title=Sandbox&action=submit

and the sanbox become an empty blank page

Checking the undefined table spacial page, I do see the Table:Companies and check the data of the table, I got the following:

Defined Fields Undefined Fields
Source Article name founded revenue
Sandbox Microsoft 1492 $8

After that, the sandbox page is not accessable any more. Or it shows as a blank page, I cannot edit/delete it

If I define the table first and then use "table" tag to save some data, the data will be saved to the table, but using "repeat" to retrive the data, the data cannot be shown on a page, again, the page shows blank.

Is wikiDB tested in mediawiki 1.9.1?

No - I haven't tested WikiDB with later versions of MW yet, but I will do so as soon as I have a bit of time and let you know how I get on right here. --HappyDog 23:15, 14 March 2007 (GMT)
OK - I've fixed this bug. As far as I know WikiDB now works on MW 1.9.1. You will need to download the latest version of WikiDB.php, but the rest of the files remain unchanged. Please let me know if you encounter any more bugs.
Here's the technical bit for those who are interested: The Parser::parse() function takes a 'clearstate' argument, which if true resets the state of the parser, and if false leaves it as it is. Because I was calling a new instance of the parser with this flag set to false, it was not being initialised properly, and so a PHP error was being raised. Presumably your system is set up to not display errors, hence the blank page. This has clearly changed from earlier versions, whether deliberately or not, which is why there was no problem on this wiki (currently running 1.5.8). Anyway, I have now set the clearstate argument to true and everything seems to work fine on both wikis (fingers crossed!) :-) --HappyDog 10:38, 29 March 2007 (BST)
Thanks, Sir! This fix the problem and it works on Mediawiki 1.9.3 as well! :)

And with mediawiki 1.10.0? Any successful test?

Hi, I'm not sure who asked this question, but I have successfully tested some of the basic functionality on my MediaWiki 1.10.0 installation. The details are here on the Tables Talk page under Creating Tables for WikiDB Extension.

Let me know if you have any questions.

--Mdrayman 05:34, 25 June 2007 (BST)

simple way to do WikiDB faster

Add second single index row_id to table wikidb_fielddata. My WikiDB is like a rocket now! --Neptus 15:06, 21 February 2007 (GMT)

Hi Neptus. Sorry about the slow response - I've been away. I've checked the table definitions and this index is already there (3rd row in the index table). Perhaps you forgot to add it when you setup your tables originally (I'm pretty sure it's been there since the start). Or maybe I've misunderstood what you are saying? --HappyDog 23:15, 14 March 2007 (GMT)
Ooops.. You are right! Then I'm a blondie... I've put it all to one index... Sorry, then your base isn't slow. It is very fast base. --Neptus 02:11, 17 March 2007 (GMT)

Count of records

Is it possible to get number of record definitions without listing all? --Neptus 21:34, 14 March 2007 (GMT)

Can you please elaborate. Do you mean via a tag (so it can be displayed in a page) or on the table definition. I'm sure it would be fairly easy to add in either case - please tell me what you would like to see and where, and I'll see what I can do. --HappyDog 23:15, 14 March 2007 (GMT)
I have 4000 definitions in my WikiDB dictonary now and it grows all the time. I can count it only using tag <repeat table...># ... </repeat> now. This listing is long.
It will be nice to have only one aktual number of positions declared in WikiDB on page (or a number of selected records), visual effect like a number of pages {{NUMBEROFARTICLES}} in wiki - simple base counter: how many records is inside. Nice gadget for information and for a boast. Then it will be a tag, I think. --Neptus 01:40, 17 March 2007 (GMT)
The problem is that it would be a single number for the whole wiki. If I added a {{NUMBEROFROWS}} magic word it would give the no. of rows in all tables in the wiki, combined. It might be possible to somehow supply an argument to a magic word (e.g. {{NUMBEROFROWS|Table:Companies}}, in which case it might be possible, but I'm not sure about that.
I can definitely add the info at the top of the table definition as a way of seeing it without having to count by hand, but that won't allow you to embed it in other articles.
I guess a parser function would be possible, e.g. {{#CountRows: Table:Companies}}. I haven't investigated them yet, but if that's the kind of thing that will do what you're after then I can look into it. --HappyDog 00:12, 27 March 2007 (BST)
Thanks. It isn't necessary for life, but if it will be not especially complicate work, it will be nice and usefull tool. --195.16.88.26 00:55, 30 March 2007 (BST)
OK - I've done a quick fix that gives the info on the 'data' tab when you're in the table namespace. I'll have a think about other ways of getting this information (suggestions welcome) but you can at least see it easily now. WikiDB.php is the only file that changed for this fix (though there may be other fixes elsewhere, depending how old your copy is). --HappyDog 17:26, 3 April 2007 (BST)
Thank you. Now I know, what big it is and what I can do with it. It's no good way for visitors, but for admin is good. --Neptus 16:04, 21 April 2007 (BST)

Table Formatting

Hi. I love this use for the Wiki, but I've had one frustration - getting formatted tables. To that end, I have made a modification for my own use, and am passing it on. Feel free to include, ignore, whatever. I just thought it might be useful. I have added two internal tags to the repeat command so that there can be footers and headers added to the output produced from the repeat tag.

The trouble was produced because something like:

{|
<repeat table="foo">
|{{{colA}}}
|{{{colB}}}
|-
</repeat>
|}

gives totally useless results, and after looking at the MediaWiki Parser.php file, it turns out that unless that is rewritten, it always will - the table parsing is all one-pass, and doesn't preserve state while in an extension.

So, I have turned that into:

{|
<repeat table="foo">
<header>
{|
</header>
|{{{colA}}}
|{{{colB}}}
|-
<footer>
|}
</footer>
</repeat>
|}

Which now gives the right results. Of course, this requires the following code I added (I know it is rudimentary, but it works...)

Change to WikiDB.php

                $Data = $Table->PerformQuery($Where, $Sort);
		if (trim($Input) == "")
			$Output = $Table->FormatTableData($Data);
		else 
		{
			$outputDef = new WikiDB_OutputFormatDef($Input);
			
			$Output = "";
			$Data = $Table->NormaliseData($Data);
//			print_r($Data);

			if($outputDef->hasHeader())
			{
				$Output .= $outputDef->getHeader() . "\n";
			}
			
			$RowFormat = $outputDef->getRow();
			foreach ($Data as $Row)
			{
				$Output .= pWikiDB_ExpandVariables($RowFormat, $Row);
			}
			
			if($outputDef->hasFooter())
			{
				$Output .= $outputDef->getFooter() . "\n";
			}
//			print($Output);
		}

Also, you must include the following file:

classWikiDB_OutputFormatDef.php

<?php

class WikiDB_OutputFormatDef 
{
	var $sHeader;
	var $sFooter;
	var $sRow;
	
	function WikiDB_OutputFormatDef($contents)
	{
		$this->sHeader = $this->stripTagItem($contents, 'header');
		$this->sFooter = $this->stripTagItem($contents, 'footer');
		$this->sRow		= $contents;
	}
	
	function hasHeader(){ return strlen($this->sHeader) > 0; }
	function hasFooter(){ return strlen($this->sFooter) > 0; }
	
	function getHeader(){ return $this->sHeader; }
	function getFooter(){ return $this->sFooter; }
	function getRow()   { return $this->sRow; }
	
	function stripTagItem(&$text, $tag)
	{
		$tagLen = strlen($tag);
		
		$start = stripos($text, '<' . $tag . '>');
		if($start == FALSE)
		{
			return '';
		}
		
		$end = stripos($text, '</' . $tag . '>');
		if($end == FALSE)
		{
			return '';
		}
		
		$tagTextStart  = $start+$tagLen+2;
		$tagTextLength = $end - $tagTextStart - 1;
		$tagTextEnd    = $end + $tagLen + 3;
		
		$tagContents = substr($text, $tagTextStart, $tagTextLength);
		//strip out the tag
		$text = substr($text, 0, $start) . substr($text, $tagTextEnd);
		
		return $tagContents;
	}
	
}  //END: Class WikiDB_OutputFormatDef

?>

Thanks!

-Jacob 15:50, 8 April 2007 (BST)

Slight Modifications

I integrated your code into my working wiki, because I too saw the uselessness of the table formatting with the repeat tag. However, I made a couple other changes:

  • the table begin and end before and after the repeat are not needed, correct? (the header and footer tags print them)
  • It seems like the table code was inserting an extra line break in the raw input, which was causing a "br" tag in the output, which was skewing the last column of each table row. To change this, I added the last line here:
if (trim($Input) == "")
  $Output = $Table->FormatTableData($Data);
else {
  $Input = trim($Input); // get rid of extra line breaks
  • However, the trim broke the tag processing in the OutputFormatDef file. On the lines "if $start == FALSE" and "if $end == false", I changed the "==" to "===" to be a more correct equality test, since after the trim those tags for me were in position 0 (== FALSE, but not ===FALSE).
  • Finally, "stripos" only works on PHP5, so I changed the 2 lines that use strpos as follows:
OLD: stripos($text, '<' . $tag . '>'));
NEW: strpos($text, strtolower('<' . $tag . '>'));

Thanks for your help and work on this. It's a cool extension.

-- Joe Clark (a guest, joeclark AT joeclarkia DOT net)

Parser Functions & Other Extensions

Due to the way that the WikiDB_Parse function was written, using a local parser it won't call into any extensions that have registered with the global parser (e.g. Parser Functions). However, I have replaced the Parse call with the following, and it seems to be working on my version 1.9.3.

function WikiDB_Parse( $Input, $Parser, $LineStart = true ) 
{
                        $title = $Parser->mTitle;
                        $options = $Parser->mOptions;
                        $output = $Parser->parse( $Input, $title, $options, true, false );
                        return $output->getText();
}

-Jacob 16:49, 14 April 2007 (BST)