Hi,
I am running QuiP against a database that has 16 levels of nesting of the same element eNest, and when I try to run the following query, the CPU just keeps spinning and when I checked the resource management panel, it’s the inosrv process that’s taking all the CPU cycles, but when I ran essentially the same query on Tamino Interactive Interface, it took only about a second to return the results.
for $e in collection(“mbench”)//eNest[@aSixtyFour=2] return
Can someone tell me why this is happening?
PS: I ran QuiP against its sample DB’s and it worked fine.
Thanks a lot!
Jazz
Could you please post your schema, your xml data and the exact query you used in the Interactive Interface.
Many thanks.
The schema is as follows:
<?xml version = "1.0" encoding = "UTF-8"?>
<xs:schema xmlns:xs = “XML Schema” xmlns:tsd = “http://namespaces.softwareag.com/tamino/TaminoSchemaDefinition”>
xs:annotation
xs:appinfo
<tsd:schemaInfo name = “bm”>
<tsd:collection name = “mbench”></tsd:collection>
<tsd:doctype name = “eNest”>
tsd:logical
tsd:contentopen</tsd:content>
</tsd:logical>
tsd:physical
tsd:structureIndexfull</tsd:structureIndex>
</tsd:physical>
</tsd:doctype>
</tsd:schemaInfo>
</xs:appinfo>
</xs:annotation>
<xs:element name = “eNest”>
xs:annotation
xs:appinfo
tsd:elementInfo
</tsd:elementInfo>
</xs:appinfo>
</xs:annotation>
<xs:complexType mixed = “true”>
<xs:choice minOccurs = “0” maxOccurs = “unbounded”>
<xs:element ref = “eNest” maxOccurs = “unbounded”></xs:element>
<xs:element ref = “eOccasional” maxOccurs = “unbounded”></xs:element>
</xs:choice>
<xs:attribute name = “aUnique1” type = “xs:integer” use = “required”>
xs:annotation
xs:appinfo
tsd:attributeInfo
tsd:physical
tsd:native
tsd:index
tsd:standard</tsd:standard>
</tsd:index>
</tsd:native>
</tsd:physical>
</tsd:attributeInfo>
</xs:appinfo>
</xs:annotation>
</xs:attribute>
<xs:attribute name = “aUnique2” type = “xs:integer” use = “required”>
xs:annotation
xs:appinfo
tsd:attributeInfo
</tsd:attributeInfo>
</xs:appinfo>
</xs:annotation>
</xs:attribute>
<xs:attribute name = “aLevel” type = “xs:integer” use = “required”>
xs:annotation
xs:appinfo
tsd:attributeInfo
tsd:physical
tsd:native
tsd:index
tsd:standard</tsd:standard>
</tsd:index>
</tsd:native>
</tsd:physical>
</tsd:attributeInfo>
</xs:appinfo>
</xs:annotation>
</xs:attribute>
<xs:attribute name = “aFour” type = “xs:integer” use = “required”>
xs:annotation
xs:appinfo
tsd:attributeInfo
tsd:physical
tsd:native
tsd:index
tsd:standard</tsd:standard>
</tsd:index>
</tsd:native>
</tsd:physical>
</tsd:attributeInfo>
</xs:appinfo>
</xs:annotation>
</xs:attribute>
<xs:attribute name = “aSixteen” type = “xs:integer” use = “required”>
xs:annotation
xs:appinfo
tsd:attributeInfo
tsd:physical
tsd:native
tsd:index
tsd:standard</tsd:standard>
</tsd:index>
</tsd:native>
</tsd:physical>
</tsd:attributeInfo>
</xs:appinfo>
</xs:annotation>
</xs:attribute>
<xs:attribute name = “aSixtyFour” type = “xs:integer” use = “required”>
xs:annotation
xs:appinfo
tsd:attributeInfo
tsd:physical
tsd:native
tsd:index
tsd:standard</tsd:standard>
</tsd:index>
</tsd:native>
</tsd:physical>
</tsd:attributeInfo>
</xs:appinfo>
</xs:annotation>
</xs:attribute>
<xs:attribute name = “aString” type = “xs:string” use = “required”>
xs:annotation
xs:appinfo
tsd:attributeInfo
tsd:physical
tsd:native
tsd:index
tsd:standard</tsd:standard>
</tsd:index>
</tsd:native>
</tsd:physical>
</tsd:attributeInfo>
</xs:appinfo>
</xs:annotation>
</xs:attribute>
</xs:complexType>
</xs:element>
<xs:element name = “eOccasional”>
xs:annotation
xs:appinfo
tsd:elementInfo
</tsd:elementInfo>
</xs:appinfo>
</xs:annotation>
xs:complexType
xs:simpleContent
<xs:extension base = “xs:string”>
<xs:attribute name = “aRef” type = “xs:integer” use = “required”>
xs:annotation
xs:appinfo
tsd:attributeInfo
tsd:physical
tsd:native
tsd:index
tsd:standard</tsd:standard>
</tsd:index>
</tsd:native>
</tsd:physical>
</tsd:attributeInfo>
</xs:appinfo>
</xs:annotation>
</xs:attribute>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:schema>
And the actual data looks something like the following:
<?xml version="1.0" encoding="utf-8"?>
ino:request
ino:object
Sing a song of oneB1
A pocket full of oneB3
Four and twenty oneB13
All baked in a oneB8.
When the oneB9 was opened,
The oneB4 began to sing;
Wasn’t that a dainty oneB1
To set before the oneB11?
The King was in his 1thirtyfiveB11,
Counting out his oneB15;
The Queen was in the oneB7
Eating bread and thirtyfourB9.
The maid was in the oneB14
Hanging out the oneB1;
When down came a oneB1,
And snipped off her 1ninetyfourB9!
Sing a song of 7seventyfourB11
A pocket full of oneB1
Four and twenty nineB7
All baked in a oneB2.
When the fortynineB7 was opened,
The 4seventyB11 began to sing;
Wasn’t that a dainty oneB10
To set before the 21fiftysixB15?
The King was in his 10seventyeightB14,
Counting out his 1eighteenB9;
The Queen was in the oneB2
Eating bread and 5fortysixB11.
The maid was in the thirtyB7
Hanging out the oneB12;
When down came a 123eightyB15,
And snipped off her 5thirtynineB13!
Sing a song of 7seventyfourB11
A pocket full of oneB1
Four and twenty nineB7
All baked in a oneB2.
When the fortynineB7 was opened,
The 4seventyB11 began to sing;
Wasn’t that a dainty oneB10
To set before the 21fiftysixB15?
The King was in his 10seventyeightB14,
Counting out his 1eighteenB9;
The Queen was in the oneB2
Eating bread and 5fortysixB11.
The maid was in the thirtyB7
Hanging out the oneB12;
When down came a 123eightyB15,
And snipped off her 5thirtynineB13!
Sing a song of 83seventysevenB15ing
A pocket full of fiftynineB8
Four and twenty eightB5
All baked in a oneB2.
When the 68thirtynineB14ing was opened,
The twoB2 began to sing;
Wasn’t that a dainty thirteenB9
To set before the fifteenB7?
The King was in his nineB5,
Counting out his 111twentythreeB15;
The Queen was in the 1seventyfourB9
Eating bread and sixtynineB8.
The maid was in the 10seventyfiveB15ing
Hanging out the oneB1;
When down came a 8ninetysevenB13,
And snipped off her 1ninetythreeB13!
</ino:object>
</ino:request>
Thanks! Any suggestion would be highly appreciated!
JAZZ
I can’t reproduce your problem. I loaded the schema and data as supplied into Tamino 3114 and 3121. Then I used Quip (Version 2.2.1.1 for Windows) to run the query you supplied. It returns nothing which seems to be correct. Changing the value of aSixtyFour to a value that is present in the data (0, 1 or 56) gives the correct results too.
What versions of Tamino and Quip are you using?
You are right, Bill. When it’s a small doc like I gave you, it doesn’t present a problem at all, but when it comes to a big doc of size 50MB, it just takes forever to run the same query…
I am using Tamino 3.1.2.1 and the latest version of QuiP. I was wondering if the old QuiP would be different, although I couldn’t find anywhere to download it. Would you happen to have a clue as to whether the versions would make a difference?
Oh, I am running QuiP on J2SDK 1.4.1.
Thanks, Bill!
Jazz
I don’t think going to an earlier version of Quip will help at all - I just wanted to make sure you were on the latest version.
If you do an ino:explain for your query you will see that the post-processor is involved (even though the attribute is indexed). I think this is because of the // xpath operator. If you try ino:explain on a simpler query such as _xql=ino:explain(eNest[@aSixtyFour=0]) the post-processor is no longer needed.
Using the post processor on 50Mb documents I would expect to be an expensive process on your server machine, placing heavy memory demands on it, and I would expect this whether the query came from Quip or from II.
However, I think that Quip uses cursoring to return your documents individually , so he issues_xql(1,1)… then _xql(2,1) and so on. In each case the entire query has to be repeated and post-processed.
So I think all you are seeing is a server very busy processing all of your 50Mb documents several times over.
[This message was edited by Bill Leeney on 14 Nov 2002 at 10:12.]
Thanks, Bill. That makes sense. So you are saying that there is virtually not too much I can do to improve the performance of QuiP on my queries? And another question, while we are at it: I noticed that there was not much difference in query response time between indexed and non-indexed databases. Essentially what is happening is that I have two databases, one with both attribute indexes and full structure index, and the other with neither of these, while performing the same query, there is not much difference in server response time… Or maybe I just didn’t set up the indexes in the right way in my schema – please refer to the schema in my previous message.
Thanks!
Jazz
It shouldn’t come as a surprise that QuiP takes a long time to process a query against a 50Mb document. I think the web site makes it fairly clear that QuiP is designed as a demonstrator for XQuery functionality, it is not an industrial strength XML database product. Even if it were, I would advise very much against putting 50Mb documents into a database - you should aim to have lots of small documents in a database rather than one big one.
It’s quite possible that QuiP has particular problems in optimizing “//” selections into a recursive structure, I don’t know anything about its internals so I wouldn’t know. But that’s not really the point: QuiP simply isn’t designed to do queries of this size, so it’s not surprising if it fails.
Michael Kay
Thanks Michael. I think I get the big picture now.
Jazz
I agree with what Michael says, but there is something to add about indexing recursive structures. Your response time problem is not caused by Quip but by the lack of an effective index to your attribute (and hence the involvement of the post-processor).
Your query does a value comparison ([@aSixtyFour=2) which means either a standard index is needed or post-processing must be done.
You have specified a standard index on your attribute, but a Tamino index always corresponds to a fixed path in the document. Your document has a recursive structure.
To make sure that every possible occurence of the attribute is supported by an index, you need to use the Schema Editor to modify the schema. If you select the attribute, and press the Advanced button in the section Physical Properties, you can specify an XPath expression to specific occurrences of your attribute, and you can index each one separately (eg eNest/eNest/@aSixtyFour and standard index).
With all of the recursions supported by a standard index, a query like eNest[eNest/eNest/@aSixtyFour=2] should be significantly faster because the post-processor should not be needed. But even with this structure, a query involving // will need to be post-processed.
Bottom line - revise your document structure as Michael suggested.
HTH
[This message was edited by Bill Leeney on 15 Nov 2002 at 10:06.]
So you are saying that if I have 16 levels of nesting then I should have 16 levels of indexing?
Jazz
I think it would be worthwhile try QuiP 2.1.1. My experience and those of others on this forum is that 2.1.1 is both faster and more accurate than 2.2.1. Our tests here involve small text documents rather than large documents stored in Tamino. But it still would probably be worth checking out for your situation.
QuiP 2.1.1 is still available for downloading from:
http://developer.softwareag.com/tamino/quip/download_old.htm
Hopefully a new version of QuiP will arrive that gets us back to the old speed and accuracy, while better tracking the latest XQuery working drafts. It has been great to have QuiP available to get familiar with XQuery’s capabilities on our own XML data!
Michael Good
www.recordare.com