This week has had a rather exciting development that several people have been wanting for quite a long time.  The 64-bit compatible versions of MG, MGPP and GDBM have been added to the main code, meaning that Greenstone 2 and 3 can now compile successfully on 64-bit systems. The reason this has taken a long time to be done is that the 32-bit and 64-bit versions of MG and MGPP produced seemingly different files when run over the same documents, which was a concerning for us as people might want to move their 32-bit MG/MGPP collections over to a 64-bit Greenstone installation and we suspected that this might not work given the different files. This week we discovered the cause of the difference and are now reassured that files from 32-bit and 64-bit installations can be interchanged without issue.

This week has seen more upgrades to Greenstone 3 as well. One of the features we have been working on for the Pei Jones collection is the ability to zoom “screen” images by using the mouse like a magnifying glass. We have added this into the default Greenstone 3 capabilities. In order for this to work however there needs to be a “screen” (small) and “source” (usually larger) version of the same image.

In general Greenstone 3 now handles paged-images much better. They are now properly displayed at the top of their specific sections. There is also an option to change between text-only, image-only and the default text and image modes, which is available in both the paged style collections as well as normal hierarchy style collections.

Next week will most likely involve more improvements like this as we continue to prepare Greenstone 3 for release.

Our exploration into security for Greenstone 3 using the built-in security provided by the Java Servlet API has gone well. We now have the ability to allow users to restrict parts of their collections to users within certain groups by specifying constraints in the collectionConfig.xml file. We are still working on the exact format for the XML but here is an example of a set of constraints in the current format:

<security scope="documents" default_access="public">
  <documentSet name="firstSet">
    <match>HASHe08571b7f6e430e238e2dd</match>
    <match field="Title" type="regex">.* Garden</match>
  </documentSet>
  <documentSet name="secondSet">
    <match field="Title" type="regex">Egyptian .*</match>
  </documentSet>
  <exception>
    <documentSet name="firstSet"/>
    <group name="dl"/>
  </exception>
  <exception>
    <documentSet name="secondSet"/>
    <group name="administrator"/>
  </exception>
</security>


You’ll notice that in the <security> element there are two attributes. The default_access attribute can be either “public” or “private” and this specifies whether the normal (guest) user can access the collection/documents. The scope attribute can be either “collection” or “documents” and this specifies whether these rules affect the whole collection or a set of documents. An average collection will have a very simple security block like:

<security scope="collection" default_access="public"/>

which specifies that the whole collection is publicly accessible. As you can see with the first example, we also allow much more detailed control over what documents each group can access. What this example specifies is that the average user can access the majority of the documents with a few exceptions. In order to access the “firstSet” set of documents (which contains the document with the ID  HASHe08571b7f6e430e238e2dd and all documents whose titles end in “Garden”) you have to be in the “dl” group. In order to access the”secondSet” set of documents (which contains documents whose titles start with “Egyptian”) you have to be an administrator.

As well as working on security I have made various improvements to the document editor. Users can now edit documents directly on the document page, which allows users to very efficiently perform any basic (content and metadata) modifications. The previous document editor is still where more advanced edits (structure, document creation/deletion) can be performed.

Things have been fairly busy here the last few weeks so I’ve been a little slack on the blogging. We have been continuing to look into robust authentication for Greenstone 3 and as part of the we have been investigating the security features that the Java Servlet technology (that Greenstone 3 uses) has built in. We have also been devising a way to specify the security settings that you want - like in Greenstone 2 - but in a way that is more flexible. For example, we are looking into the idea of groups of users (e.g. admin, staff, students etc.) that can have access to different documents based on the groups they are in.

I’ll write more details on this next week.

This week I have been touching up on a few unfinished features. One of these was the mapping features that I can’t remember whether or not I have written about before. Basically, if your documents have coordinate information (i.e. latitude and longitude information) we now have a feature that will map those documents on a map. This feature can now be really easily enabled. We will write some documentation on this when we get the chance.

The new theme is also added which makes Greenstone 3 look a lot nicer. We’re still working on the ability to allow easy theme changing. We need to get the authentication working before we can enable this feature, as only the collection administrator should be able to change the theme.

Next week I will experimenting with trying to get more standard URLs in Greenstone 3 (e.g. http://localhost:8383/greenstone3/dev/collection/demo/document/HASHc5bce2d6d3e5b04e470ec8) rather than what we currently use.

P.S. If you’re wondering why there is no update from Anu this week, it is because she is away in India for 5 weeks on holiday.

One of the things we have been doing this week is deciding the best way to handle user authentication in Greenstone 3. We have a very basic system in place at the moment but we would like something more robust. At the moment we are investigating using the authentication system in the web-server we use for Greenstone 3 (Apache Tomcat). We need to make sure it has the flexibility we require so that collection administrators have the power to allow/prevent users access to the collection as well as (possibly) access to individual documents.

I have been continuing to assist the masters student I mentioned last week. We have been working on a way to download and replace parts of a collection via the web interface. We think that this functionality may be useful if you want to add/replace an image or run an image through your own OCR program for example.

Finally, I have been further adding to Greenstone’s CGI metadata capabilites, filling in any holes that are missing in the API. As part of this I have started developing a Javascript API which should (theoretically) make using these CGI calls a lot easier.

My time this week has mostly been spent helping out one the masters students here in our lab. I have been helping her develop the ability to tag photos and text in the Greenstone 3 collection she is working on. This has resulted in us enhancing our Greenstone 3 (and also Greenstone 2) CGI capabilities at the same time to get this working correctly. This upgrade was needed so that we could save metadata to the index, archive and import directories easily from Javascript. Some of the functionality was already there but functionality like the ability to remove metadata from the import directory (for example) was missing.

One problem we had to get around was the fact that you cannot reliably specify the position of a piece of metadata that you want to change/delete in a metadata.xml file because of the way import metadata is handled in Greenstone. We decided that a good way to get around this is to have to specify the previous value of the piece of metadata that you want to change/delete. The only problem with this approach is if you have more than one identical piece of metadata, do we delete just one? or all of them? Most likely we will add an option to specify what to do in this situation.

Next week I will most likely be working on some authentication functionality for Greenstone3.

Happy new year to all Greenstone users! We’re back at work now after a couple of weeks off over the holiday period and already we’ve got a few new things lined up.

In Greenstone we try very hard to make the modification of the look and feel of collections as easy as possible.  Unfortunately this often requires knowledge of web standards like HTML, CSS and Javascript, and in the case of Greenstone 3 it is also helpful to have knowledge of XML and XSLT. We understand that many Greenstone users will have very little knowledge of these topics, so we are looking at incorporating a very simple way of changing the appearance of a Greenstone collection.

JQuery UI has a system called ThemeRoller that allows you to create your own visual theme via an easy to use web interface. You can then download the required files to use that theme in your own website. We are currently experimenting with making Greenstone 3 compatible with these themes (which are made up of a CSS file and some images). So far it is looking promising and will hopefully prove to be a welcome addition to Greenstone 3.

It has been a short week this week so there’s not a lot to report, but next week I shall be continuing on this development as well as (most likely) starting to write some up-to-date documentation for Greenstone 3, as we have made it our goal this summer to spend a large part of it working on Greenstone’s documentation.

This week I have been tidying up the new paged-image functionality so that it dynamically loads each page (rather than doing a full page reload each time) and also added the functionality that allows the user to choose from “Text view” (which only shows the OCR’d text), “Image view” (which shows the original image) and “Default view” (which shows both the text and the image). These are also switched dynamically which is nice and are remembered if you leave a document page and go to a new one.

I also fixed up an annoying problem with GLI. One of the ways you can customise collections in Greenstone 3 is by writing Javascript in the collectionConfig.xml file and those familiar with XML will know that you cannot put ‘&’, ‘<’ or ‘>’ into text nodes (you have to replace them with &amp;, &lt; and &gt; respectively). These special characters a relatively common in Javascript so each time they are used they have to be escaped. The problem we were having with GLI was that it would read in the file and replace the characters with their usual forms (&, < and >) and when it went to save the file it wouldn’t escape these characters. So the next time this file was read in GLI would produce an error because the file was no longer valid XML. We eventually tracked this problem down and fixed it.

Next week I will continue to work on the paged-image functionality (specifically the “next page” and “previous page” buttons) as well as adding some new code to HTMLPlugin that will add any files referred to in CSS files (e.g. background-image) as associated files of the HTML page.

This week has mostly been spent improving Greenstone 3’s capability to display paged documents. This has mostly involved upgrading the table of contents functionality to better handle documents with a lot of pages and also have names like “Page 1″, “Page 2″, “Page 3″ etc. making them virtually indistinguishable by their names. In this case it would be much better if images of the pages were displayed. Fortunately many of these collections will already have these thumbnails available so these will now be displayed in the table of contents instead of their names. Simply replacing the names with images however results in two more problems. The first is that a lot of images take up a lot of space on the page, and the second problem is that it greatly increases the amount that the user has to download from the server for each page. Even though a single black and white thumbnail is likely only to be around 10KB in size, having a thousand of these (which is not unrealistic), or if the images are color then they can quickly add up in size.

To solve both of these problems I decided that a good option would be to create a box in the table of contents that only shows a few pages at a time and can be scrolled from right to left to go through the images of the pages. As well as saving space, this approach also has the added benefit that images do not need to be loaded until they are visible within the box (i.e. they have been scrolled over). So I have implemented it so that images are loaded dynamically as necessary.

I have also added a new feature to Greenstone 3 that may prove useful in improving some of the interactions that happen between XSLT and Javascript. One thing I have been needing to do a reasonable amount recently is take parts of pages and add them to other pages. Our current method for doing this is to get the page we want and to “cut” the detail we want out of it. To hopefully smooth out this interaction I have added the ability for XSL templates to be specified in the CGI arguments given to the page. This allows Javascript AJAX calls to single out the exact part of the page they want or even create new information, all in a single AJAX call.

This week has mostly been focused on bug fixing. One bug we discovered a while ago was that the code that highlights search terms in the text would also find occurrences of the terms inside tags (e.g. it would find the word farming in <a href=”farming.html”>farming</a>). The fix was to exclude the characters inside these tags from being considered by the highlight searching code by looking for the < character and ignoring all characters until we see a > character. You may be thinking “But what if there is a < in the document text?”, the answer is that this isn’t an issue as the document text will not contain any of these characters that don’t belong to tags as they will be escaped as &lt; and &gt;.

Another bug I fixed was to do with the Document Structure Editor. The bug was that it always wiped the contents of any images in the collection that was being built, leaving empty files, but the XML files were being preserved fine. The main bug was caused by the index directory not being deleted correctly. This was because the server still had the collection loaded in the runtime system (so that it can be viewed) while it tried to delete its index. So it required that the collection be briefly deactivated in the runtime system so that this replacement (the newly built index replacing the old one) could take place.

Another problem was with displaying paged-image collections. The system would only ever show the root level section and the top level sections and no sections lower than that. I tracked this down to the top levels sections being marked as “leaf” nodes instead of “internal” nodes. Whether this is a bug or whether this has been done deliberately I will try and figure out next week.

Also next week I will do some work on enabling a basic form of spatial searching (searching by locations) in any collections that contain documents with latitude and longitude information.