Difference between revisions of "Cache API"

From FollowTheScore
Jump to: navigation, search
 
(7 intermediate revisions by the same user not shown)
Line 7: Line 7:
  
  
−
Updated : [[User:EmuWikiAdmin-|EmuWikiAdmin-]] 04:43, 1 July 2009 (UTC)
+
Updated : [[User:EmuWikiAdmin-|EmuWikiAdmin-]] 05:57, 1 July 2009 (UTC)
  
 +
 +
'''Installing'''
 +
----
  
 
For now this SQL command needs to be executed manually before installing :
 
For now this SQL command needs to be executed manually before installing :
Line 14: Line 17:
 
<pre>
 
<pre>
 
  CREATE TABLE IF NOT EXISTS `cacheapi` (
 
  CREATE TABLE IF NOT EXISTS `cacheapi` (
 +
`dependencies` TEXT CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
 +
`firstdep` TEXT CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
 +
`types` TEXT CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
 +
`firsttype` SMALLINT NOT NULL ,
 
`page_ids` INT(10) NOT NULL ,
 
`page_ids` INT(10) NOT NULL ,
−
`dependencies` TEXT CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
+
INDEX ( `firstdep`(30) ) ,
−
`first` TEXT CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
+
INDEX ( `firsttype` ) ,
−
`type` INT(10) NOT NULL ,
+
INDEX ( `page_ids` )
−
INDEX ( `page_ids` , `dependencies`(40) , `first`(40) , `type` )
 
 
) ENGINE = MYISAM  
 
) ENGINE = MYISAM  
 
</pre>
 
</pre>
 +
  
 
'''Removing dependencies'''
 
'''Removing dependencies'''
 
----
 
----
  
−
To remove all dependencies of a page, use CacheAPI::remDependencies ( idOfThePage ).
+
To remove all dependencies of a page, use CacheAPI::remDependencies ( $idOfThePage ).
 +
 
  
  
Line 31: Line 39:
 
----
 
----
  
−
To add a dependency, use CacheAPI::addDependencies ( pageid , type, condition, '');
+
To add a dependency, use CacheAPI::addDependencies ( $pageid , $types, $conditions );
  
−
type 1 = category, 2 = template, 3 = linksto
+
$types is an array that needs to be as long as the first dimension of the 2-dimensional array $conditions. Values in $types can be :
  
−
condition is an array of category names (to be switched to an array of category numbers, see discussion below).
+
<pre>
 +
CACHETYPE_CATEGORY
 +
CACHETYPE_TEMPLATE
 +
CACHETYPE_LINKSTO
 +
CACHETYPE_NOTCATEGORY
 +
CACHETYPE_NOTTEMPLATE
 +
CACHETYPE_NOTLINKSTO
 +
CACHETYPE_LINKSFROM
 +
CACHETYPE_NAMESPACE
 +
CACHETYPE_NOTNAMESPACE
 +
CACHETYPE_OPENREFERENCES
 +
CACHETYPE_LINKSTOEXTERNAL
 +
CACHETYPE_IMAGEUSED
 +
CACHETYPE_IMAGECONTAINER
 +
CACHETYPE_TEMPLATESUSEDBY
 +
CACHETYPE_CREATEDBY
 +
CACHETYPE_NOTCREATEDBY
 +
CACHETYPE_MODIFIEDBY
 +
CACHETYPE_NOTMODIFIEDBY
 +
CACHETYPE_LASTMODIFIEDBY
 +
CACHETYPE_NOTLASTMODIFIEDBY
 +
CACHETYPE_TITLE
 +
CACHETYPE_TITLESHORTERTHAN
 +
CACHETYPE_TITLELONGERTHAN
 +
CACHETYPE_LASTREVISIONBEFORE
 +
CACHETYPE_FIRSTREVISIONSINCE
 +
CACHETYPE_ALLREVISIONBEFORE
 +
CACHETYPE_ALLREVISIONSINCE
 +
CACHETYPE_MAXREVISIONS
 +
CACHETYPE_MINREVISIONS
 +
CACHETYPE_ARTICLECATEGORY
  
−
----
+
CACHETYPE_ORCATEGORY
−
TODO :
+
CACHETYPE_ORTEMPLATE
 +
CACHETYPE_ORLINKSTO
 +
CACHETYPE_ORNOTCATEGORY
 +
CACHETYPE_ORNOTTEMPLATE
 +
CACHETYPE_ORNOTLINKSTO
 +
CACHETYPE_ORLINKSFROM
 +
CACHETYPE_ORNAMESPACE
 +
CACHETYPE_ORNOTNAMESPACE
 +
CACHETYPE_OROPENREFERENCES
 +
CACHETYPE_ORLINKSTOEXTERNAL
 +
CACHETYPE_ORIMAGEUSED
 +
CACHETYPE_ORIMAGECONTAINER
 +
CACHETYPE_ORTEMPLATESUSEDBY
 +
CACHETYPE_ORCREATEDBY
 +
CACHETYPE_ORNOTCREATEDBY
 +
CACHETYPE_ORMODIFIEDBY
 +
CACHETYPE_ORNOTMODIFIEDBY
 +
CACHETYPE_ORLASTMODIFIEDBY
 +
CACHETYPE_ORNOTLASTMODIFIEDBY
 +
CACHETYPE_ORTITLE
 +
CACHETYPE_ORTITLESHORTERTHAN
 +
CACHETYPE_ORTITLELONGERTHAN
 +
CACHETYPE_ORLASTREVISIONBEFORE
 +
CACHETYPE_ORFIRSTREVISIONSINCE
 +
CACHETYPE_ORALLREVISIONBEFORE
 +
CACHETYPE_ORALLREVISIONSINCE
 +
CACHETYPE_ORMAXREVISIONS
 +
CACHETYPE_ORMINREVISIONS
 +
CACHETYPE_ORARTICLECATEGORY
 +
</pre>
  
−
* Implement other types of dependencies (Namespace, linksfrom, etc...)
+
For now, only Categories are fully implemented for testing but the rest is implemented as dummy and should be fully ready very soon. To avoid performance issues, we will restrict the possible first types in your array. For example, a NOT should not be used as your first type, just as 2nd or more...
−
* Implement the NOTs (notcategory, notlinksto, etc...)
 
−
* Make a better system for deleation of dependencies (right now, the only thing you can do is delete ALL dependencies related to a page at once)
 
−
* Implement cache outdating when special events other than edit/create are performed (deletion, rename, etc...)
 
−
* If you need anything else, add it here.
 
  
−
== First Feedback ==
+
$condition is a bidimensional array of category names, template names, title names, etc...
−
Hello Jean-Francois,
 
  
−
I decided to call your API immediately before executing my own SQL, that is near line 2418 in DPLMain.php.
 
−
I used the following code to call your interface:
 
  
−
<pre>
 
  
−
// update dependencies to CacheAPI whenever the page containing the DPL query is edited
+
'''Example'''
−
+
----
−
if ($wgRequest->getVal('action','view')=='submit') {
 
−
CacheAPI::remDependencies ( $wgArticle->getID());
 
−
foreach ($aIncludeCategories as $categoryGroup) {
 
−
foreach ($categoryGroup as $category) {
 
−
$title = Title::makeTitle(14, $category);
 
−
$catID = $title->getArticleID();
 
−
CacheAPI::addDependencies ( $wgArticle->getID(), 1, $catID, '');
 
−
// die ("adding to DEPENDENCIES: ".$title->getArticleID());
 
−
}
 
−
}
 
−
}
 
−
</pre>
 
  
−
* you must add a "global $wgArticle;" somwehere above this code.
+
'''Example with only ANDs'''
−
::Ok tried it. You version didn't work because you were using numbers (page ids) for categories, I was using text. See discussion below.
 
−
* Is it correct that you expect the pageID of the page containing the DPL statement as the first argument?
 
−
::Yes.
 
−
* Would it make sense to add a second index to your table to spped up search when dependant articles are changed?
 
−
::Yes, all columns will be indexed. The SQL query makes them all indexes. Decisions remains to be taken whether or not we use page ids or text as dependencies.
 
−
* What is the & separator good for? I have groups of OR-wired cats. All groups are AND-wired....
 
−
::Aren't we forced to stored the AND relations between the categories ? For example, When a DPL invocation looks for articles in categories AA & BB, we don't want to outdate the cache of the DPL page's invocation if an article of only category AA is touched. We want it to be purged only if an article of AA _AND_ BB is touched. The way you used addDependencies should be used only for the OR relation, that is to add separate entries for separate categories. When the AND relation is present (specified by user or implicit), you should pass an array of category names as a 3rd argument.
 
−
* I would like to use symbolic constants instead of pure numbers for the types (1,2,3)
 
−
::Ok. In the next version, we will use CACHE_CATEGORYTYPE CACHE_TEMPLATETYPE and CACHE_LINKSTOTYPE
 
−
* I had to delete the final php delimiter at the end of your source code because there seems to be some invisible UTF code after it.
 
−
::No idea how it happened! Probably my text editor/uploader.
 
−
* I created a small article containing the following query:
 
  
−
<pre>
 
−
        {{#time:Y-m-d h:i:s}}
 
−
        {{#dpl:
 
−
        | category = Test¦Fictitious country
 
−
        }}
 
−
</pre>
 
  
−
* I used the following parameter in the LocalSettings.php (because this would be a typical configuration for a huge wiki I guess):
+
Say we have a DPL invocation like this :
−
  ExtDynamicPageList::$respectParserCache = true;
 
−
* Instead of doing this I could have written in the DPL query
 
−
| allowcachedresults=true
 
−
* The #time statement is very useful - so you can see that the time does NOT change if you reload the query page multiple times.
 
−
* After editing the query page I saw two entries in the new table, correctly pointing to the two categories (Test , Fictitious country)
 
−
::The only reason it worked for you is that all your categories are true pages that are in fact created. This is not the case for all wikis and probably not for Wikipedia. We have to find another way to work with categories than the makeTitle strategy using page numbers to identify them. See discussion below.
 
−
* Multiple reloading of the query page showed the same date/time each time (which is correct because the ParserCache is enabled)
 
−
* Then I modified [[Nigunda Test]] which is one of the articles occuring in the query result in a different browser window.
 
−
::I will send you a modified version of both my extension and your DPLMain.php that work.
 
−
* I expected the cache now to be invalidated. So when I pressed F5 on the window with the query page I should have gotten a new time stamp - but this did not happen.
 
  
−
* I have no idea how the ParserCache works but the whole idea is:
 
−
*# use the ParserCache - even for pages containing DPL queries (by default DPL switches the cache off so onbe must pay attention here)
 
−
*# change a page contained in the result (regardless what kind of change, I assume at the moment)
 
−
*# refresh the DPL page
 
−
*# ... and watch that the DPL page is no longer delivered from cache but recalculated.
 
  
−
I think it is best if you set up a similar configuration (article names and cat names may differ) and then try to make the simple example with one or two categories work on your pc. If you have to change the lines in DPLMain.php which I used to call your API: go ahead and tell me what you have done.
+
<pre>
 +
{{#dpl:
 +
|allowcachedresults=yes
 +
|category = Test&Blablou&Papou
 +
|notcategory = Mimi
 +
|nottemplate = Stars
 +
|include            ={Test}:String
 +
|format              =,,,
 +
|secseparators      =,%PAGE%,
 +
}}
 +
</pre>
  
 +
Then the CacheAPI should be called with the following parameters :
  
−
Note: I observed that your table changes also if a dependant page is being edited. In my case another entry was added for 'Nigunda Test'. I do not understand why this is necessary and I am afraid it will slow down regular editing.
+
<pre>
−
::This is weird. Was there a DPL invocation in Nigunda Test ? I've never seen this behavior. We'll look at this when you try the corrected versions.
+
$types = array ( CACHETYPE_CATEGORY , CACHETYPE_NOTCATEGORY , CACHETYPE_NOTTEMPLATE );
  
 +
$conditions = array (
 +
                    array ( 'Test' , 'Blablou' , 'Papou' ) ,
 +
                    array ( 'Mimi' ) ,
 +
                    array ( 'Stars' )
 +
              );
  
−
Happy testing!
+
CacheAPI::addDependencies ( $pageid , $types, $conditions );
 +
</pre>
  
−
Gero
 
  
−
----
+
'''Example with ORs'''
  
−
<s>I'll take the time to review each of your remarks, thanks a lot for testing. However I think there's an error in the way you use addDependencies, and i'd like you to check this. The third argument seems to be a problem, I was expecting a string containing the conditional statement with '&' as separators. For example, if you want page 605 (which contain the DPL statement) to be dependant on the edition of articles that are in categories AA & BB, I would expect a call like this :
 
  
−
CacheAPI::addDependencies ( $wgArticle->getID(), 1, 'AA&BB', '');
+
Say we have a DPL invocation like this :
  
−
or if it's just dependent on 1 category like AA :
 
  
−
CacheAPI::addDependencies ( $wgArticle->getID(), 1, 'AA', '');</s> - outdated, the 3rd argument for now will be an array of category names. Could be an array of category numbers too, see below.
+
<pre>
 +
{{#dpl:
 +
|allowcachedresults=yes
 +
|category = Test|Blablou
 +
|notcategory = Mimi
 +
|nottemplate = Stars
 +
|include            ={Test}:String
 +
|format              =,,,
 +
|secseparators      =,%PAGE%,
 +
}}
 +
</pre>
  
 +
Then the CacheAPI should be called 2 times with the following parameters :
  
 +
<pre>
 +
$types = array ( CACHETYPE_CATEGORY , CACHETYPE_NOTCATEGORY , CACHETYPE_NOTTEMPLATE );
  
−
Actually the way you did it might just be better, I might switch to using page numbers instead of the names of the categories... I don't know what happens with mediawiki when category articles are deleted and recreated. I'll have to look at this, we don't want category numbers stored in the cacheapi to mean nothing whenever a category is changed, or deleted and recreated after that. For this reason I used the category names. I'd still like to see if it works for your test if you call addDependencies as described above. [[User:EmuWikiAdmin-|EmuWikiAdmin-]] 02:14, 1 July 2009 (UTC)
+
$conditions = array (
 +
                    array ( 'Test' ) ,
 +
                    array ( 'Mimi' ) ,
 +
                    array ( 'Stars' )
 +
              );
  
−
I just tested it. Did you know that Title::makeTitle(14, $category) does not work if the category page has not been created ? This is due to the fact that a category can exist without the category page being made. That's a big problem because lots of categories are likely to exist but not all of them
+
CacheAPI::addDependencies ( $pageid , $types, $conditions );
−
have a category page already made. Maybe we should go back to storing the dependencies in the form of strings like AA&BB, or do you see another possibility ? Maybe we should work with cat_ids instead of page_ids. Look at the Category table, it might be easier to index and it would speed up the search, and it wouldnt have the same problems that makeTitle has. [[User:EmuWikiAdmin-|EmuWikiAdmin-]] 04:43, 1 July 2009 (UTC)
 
  
−
Here's a working version of the Cache API : http://www.emuwiki.com/testinstall/extensions/CacheAPI_current.txt . Right now I'll go on using text to store categories dependencies because of the problem discussed above regarding page IDs. We'll see if we find another solution but you can try this for now. The difference with this version is that CacheAPI::addDependencies now takes an array of category names as 3rd argument. Here's what you will need to use in DPLMain.php :
 
  
−
<pre>
+
$types = array ( CACHETYPE_CATEGORY , CACHETYPE_NOTCATEGORY , CACHETYPE_NOTTEMPLATE );
  
−
// update dependencies to CacheAPI whenever the page containing the DPL query is edited
+
$conditions = array (
−
+
                    array ( 'Blablou' ) ,
−
if ($wgRequest->getVal('action','view')=='submit') {
+
                    array ( 'Mimi' ) ,
−
CacheAPI::remDependencies ( $wgArticle->getID());
+
                    array ( 'Stars' )  
−
foreach ($aIncludeCategories as $categoryGroup) {
+
              );
−
CacheAPI::addDependencies ( $wgArticle->getID(), 1, $categoryGroup, '');  
 
−
}
 
−
}
 
  
 +
CacheAPI::addDependencies ( $pageid , $types, $conditions );
 
</pre>
 
</pre>
  
−
[[User:EmuWikiAdmin-|EmuWikiAdmin-]] 04:43, 1 July 2009 (UTC)
+
----
 +
TODO :  
  
 +
* Implement other types of dependencies (Namespace, linksfrom, etc...) - currently being done
 +
* Implement the NOTs (notcategory, notlinksto, etc...) - currently being done
 +
* Make a better system for deletion of dependencies (right now, the only thing you can do is delete ALL dependencies related to a page at once, this is not good for pages with multiple DPL invocations).
 +
* If you need anything else, add it here.
 +
----
  
−
You can try for yourself, but right now I need no more input from you I'll go on and improve the system, since I know where you call the API in DPL I might also change DPLMain.php itself if I have problems. I'll be back with more functionalities and a better manual to document all those functionalities. Thanks for your test! [[User:EmuWikiAdmin-|EmuWikiAdmin-]] 04:53, 1 July 2009 (UTC)
+
Feedback and discussion moved [[Talk:Cache API|here]].

Latest revision as of 16:31, 1 August 2009

This is a development page for the Cache API that we are currently implementing. Talk to EmuWikiAdmin- for more information.

Manual

We have a somewhat working version here : http://www.emuwiki.com/testinstall/extensions/CacheAPI_current.txt


Updated : EmuWikiAdmin- 05:57, 1 July 2009 (UTC)


Installing

For now this SQL command needs to be executed manually before installing : 

 CREATE TABLE IF NOT EXISTS `cacheapi` (
`dependencies` TEXT CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`firstdep` TEXT CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`types` TEXT CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`firsttype` SMALLINT NOT NULL ,
`page_ids` INT(10) NOT NULL ,
INDEX ( `firstdep`(30) ) ,
INDEX ( `firsttype` ) ,
INDEX ( `page_ids` )
) ENGINE = MYISAM


Removing dependencies

To remove all dependencies of a page, use CacheAPI::remDependencies ( $idOfThePage ).


Adding dependencies

To add a dependency, use CacheAPI::addDependencies ( $pageid , $types, $conditions );

$types is an array that needs to be as long as the first dimension of the 2-dimensional array $conditions. Values in $types can be : 

CACHETYPE_CATEGORY
CACHETYPE_TEMPLATE
CACHETYPE_LINKSTO
CACHETYPE_NOTCATEGORY
CACHETYPE_NOTTEMPLATE
CACHETYPE_NOTLINKSTO
CACHETYPE_LINKSFROM
CACHETYPE_NAMESPACE
CACHETYPE_NOTNAMESPACE
CACHETYPE_OPENREFERENCES
CACHETYPE_LINKSTOEXTERNAL
CACHETYPE_IMAGEUSED
CACHETYPE_IMAGECONTAINER
CACHETYPE_TEMPLATESUSEDBY
CACHETYPE_CREATEDBY
CACHETYPE_NOTCREATEDBY
CACHETYPE_MODIFIEDBY
CACHETYPE_NOTMODIFIEDBY
CACHETYPE_LASTMODIFIEDBY
CACHETYPE_NOTLASTMODIFIEDBY
CACHETYPE_TITLE
CACHETYPE_TITLESHORTERTHAN
CACHETYPE_TITLELONGERTHAN
CACHETYPE_LASTREVISIONBEFORE
CACHETYPE_FIRSTREVISIONSINCE
CACHETYPE_ALLREVISIONBEFORE
CACHETYPE_ALLREVISIONSINCE
CACHETYPE_MAXREVISIONS
CACHETYPE_MINREVISIONS
CACHETYPE_ARTICLECATEGORY

CACHETYPE_ORCATEGORY
CACHETYPE_ORTEMPLATE
CACHETYPE_ORLINKSTO
CACHETYPE_ORNOTCATEGORY
CACHETYPE_ORNOTTEMPLATE
CACHETYPE_ORNOTLINKSTO
CACHETYPE_ORLINKSFROM
CACHETYPE_ORNAMESPACE
CACHETYPE_ORNOTNAMESPACE
CACHETYPE_OROPENREFERENCES
CACHETYPE_ORLINKSTOEXTERNAL
CACHETYPE_ORIMAGEUSED
CACHETYPE_ORIMAGECONTAINER
CACHETYPE_ORTEMPLATESUSEDBY
CACHETYPE_ORCREATEDBY
CACHETYPE_ORNOTCREATEDBY
CACHETYPE_ORMODIFIEDBY
CACHETYPE_ORNOTMODIFIEDBY
CACHETYPE_ORLASTMODIFIEDBY
CACHETYPE_ORNOTLASTMODIFIEDBY
CACHETYPE_ORTITLE
CACHETYPE_ORTITLESHORTERTHAN
CACHETYPE_ORTITLELONGERTHAN
CACHETYPE_ORLASTREVISIONBEFORE
CACHETYPE_ORFIRSTREVISIONSINCE
CACHETYPE_ORALLREVISIONBEFORE
CACHETYPE_ORALLREVISIONSINCE
CACHETYPE_ORMAXREVISIONS
CACHETYPE_ORMINREVISIONS
CACHETYPE_ORARTICLECATEGORY

For now, only Categories are fully implemented for testing but the rest is implemented as dummy and should be fully ready very soon. To avoid performance issues, we will restrict the possible first types in your array. For example, a NOT should not be used as your first type, just as 2nd or more...

$condition is a bidimensional array of category names, template names, title names, etc...


Example

Example with only ANDs


Say we have a DPL invocation like this : 


{{#dpl:
|allowcachedresults=yes
|category = Test&Blablou&Papou
|notcategory = Mimi
|nottemplate = Stars
|include             ={Test}:String
|format              =,,,
|secseparators       =,%PAGE%,
}}

Then the CacheAPI should be called with the following parameters : 

$types = array ( CACHETYPE_CATEGORY , CACHETYPE_NOTCATEGORY , CACHETYPE_NOTTEMPLATE );

$conditions = array (
                    array ( 'Test' , 'Blablou' , 'Papou' ) ,
                    array ( 'Mimi' ) ,
                    array ( 'Stars' ) 
              );

CacheAPI::addDependencies ( $pageid , $types, $conditions );


Example with ORs


Say we have a DPL invocation like this : 


{{#dpl:
|allowcachedresults=yes
|category = Test|Blablou
|notcategory = Mimi
|nottemplate = Stars
|include             ={Test}:String
|format              =,,,
|secseparators       =,%PAGE%,
}}

Then the CacheAPI should be called 2 times with the following parameters : 

$types = array ( CACHETYPE_CATEGORY , CACHETYPE_NOTCATEGORY , CACHETYPE_NOTTEMPLATE );

$conditions = array (
                    array ( 'Test' ) ,
                    array ( 'Mimi' ) ,
                    array ( 'Stars' ) 
              );

CacheAPI::addDependencies ( $pageid , $types, $conditions );


$types = array ( CACHETYPE_CATEGORY , CACHETYPE_NOTCATEGORY , CACHETYPE_NOTTEMPLATE );

$conditions = array (
                    array ( 'Blablou' ) ,
                    array ( 'Mimi' ) ,
                    array ( 'Stars' ) 
              );

CacheAPI::addDependencies ( $pageid , $types, $conditions );

TODO :

  • Implement other types of dependencies (Namespace, linksfrom, etc...) - currently being done
  • Implement the NOTs (notcategory, notlinksto, etc...) - currently being done
  • Make a better system for deletion of dependencies (right now, the only thing you can do is delete ALL dependencies related to a page at once, this is not good for pages with multiple DPL invocations).
  • If you need anything else, add it here.

Feedback and discussion moved here.

Retrieved from "http://followthescore.org/dpldemo/index.php?title=Cache_API&oldid=13496"