Ticket #51 (new enhancement)

Opened 6 years ago

Last modified 4 years ago

Requests cache

Reported by: djay Owned by:
Priority: major Milestone:
Component: zoo-kernel Version:
Keywords: Cc:

Description

When running several Services using the same inputs xlink:href value, ZOO-Kernel download again and again the same resulting data. It should be better to get an internal cache mechanism which permit the ZOO-Kernel to reuse previously downloaded data.

Change History

Changed 6 years ago by djay

A primitive cache mechanism was introduced in ZOO-Kernel in rev.280.

Note that per default the cache system is not used. To activate it, you should add a cacheDir key in your [main] section getting as value the full path to the cache directory.

Let suppose that your local cache directory is: /var/zoo_cache/, then you should get the following in the [main] section of your main.cfg file:

cacheDir = /var/zoo_cache

Obviously, you should take care about user privileges for this directory which should be writable by your apache user.

Once the cache system is activated, ZOO-Kernel will store on your disk some of the downloaded files used as input by your ZOO-Services. By now, POST requests resulting file are not stored on disk, only simple GET requests are (for instance simple GetFeature requests).

Now, over the time ZOO-Kernel will create and manage a file named list which will be used to store pairs containing: the url of the remote data and the local file containing the downloaded result. The url and the file name are separated by a "|" character. You can find in the following an example of line you can find in your list file:

http://www.zoo-project.org:8082/geoserver/ows?SERVICE=WFS&REQUEST=GetFeature&VERSION=1.0.0&typename=topp:states&SRS=EPSG:4326&FeatureID=states.15|/var/zoo_cache/0.zca

To remove the cache content, you can remove the list file or / and associated .zca files.

Note that with rev.280 there is no way to limit size of the list file or of the directory pointed by the cacheDir key. There is also no way to limit individual cached file size or number of stored files. So cache have to be cleaned periodically by external tool.

We shall think of adding new keys in the main.cfg for this kind of specific parameters and maybe even a [cache] section.

Changed 6 years ago by djay

  • component changed from Development platform to zoo-kernel

Changed 6 years ago by djay

  • milestone changed from 1.2.0 to 1.3dev

It should be useful to implement the cache system using libmemcache to enhance search speed and support time to live of cached requests.

The idea is the same as the current implementation for caching requests, so as follow, but using the memcache client API. Note that we also added the md5 calculation to name the key rather than using the full url, this way less characters have to be compared, this have to be added in the current cache implementation.

 get the md5 key of the GET request (using the url) or POST (using the XML request+the url)
 if md5 key isInCache:
    return the content in cache
 else:
   store the content in cache
   return the stored content

I set the milestone to 1.3dev as the memcache support should not be added in 1.2.0 version.

Changed 4 years ago by djay

  • milestone 1.3dev deleted

Milestone 1.3dev deleted

Note: See TracTickets for help on using tickets.

Search

Context Navigation

ZOO Sponsors

http://www.zoo-project.org/trac/chrome/site/img/geolabs-logo.pnghttp://www.zoo-project.org/trac/chrome/site/img/neogeo-logo.png http://www.zoo-project.org/trac/chrome/site/img/apptech-logo.png http://www.zoo-project.org/trac/chrome/site/img/3liz-logo.png http://www.zoo-project.org/trac/chrome/site/img/gateway-logo.png

Become a sponsor !

Knowledge partners

http://www.zoo-project.org/trac/chrome/site/img/ocu-logo.png http://www.zoo-project.org/trac/chrome/site/img/gucas-logo.png http://www.zoo-project.org/trac/chrome/site/img/polimi-logo.png http://www.zoo-project.org/trac/chrome/site/img/fem-logo.png http://www.zoo-project.org/trac/chrome/site/img/supsi-logo.png http://www.zoo-project.org/trac/chrome/site/img/cumtb-logo.png

Become a knowledge partner

Related links

http://zoo-project.org/img/ogclogo.png http://zoo-project.org/img/osgeologo.png