Many XQuery Updates launched in threads (to gain time)

, ,
1

hello all,

As i noticed that launching one xquery update on only one element doesn’t take much time (between 200 and 600 ms on my machine) i am trying to make my java client launch many updates at the time in encapsulating every single update in a thread (every single thread using a pooledConnection delivered by one general poolManager, see code below).

Doing so, i have noticed that instead of gaining time, every single thread takes more and more time to complete its run() method.
Am i misunderstanding the way Tamino handles pooled Connections ? Am i doing something wrong ?

Creation of the poolmanager and its descriptor :

 
	public ProcessTEI2(String databaseURI, String collection, String mode) throws TException, Exception { 
		 
		this.collection = collection;  
		 
		if (poolManager==null) 
			poolManager = TConnectionPoolManager.getInstance(); 
				 
		TConnectionPoolDescriptor descriptor = new TConnectionPoolDescriptor(); 
		descriptor.setDatabaseURI(databaseURI); 
		descriptor.setUser("sag"); 
		descriptor.setPassword("sag"); 
		descriptor.setMaxConnections(100); 
		descriptor.setInitConnections(50); 
		descriptor.setTimeOut(30); 
		 
		poolManager.addConnectionPool(POOL_NAME, descriptor); 
	}

Launching of one thread per xquery

 
	public void performBatchUpdateXQuery(ArrayList batchQueries, JProgressBar pb, String mode) throws TException  { 
			final TXMLObjectModel dom4jObjectModel = TDOM4JObjectModel.getInstance(); 
			TXMLObjectModel.register( dom4jObjectModel ); 
 
			for (Iterator it = batchQueries.iterator(); it.hasNext();){ 
				final TXQuery query = (TXQuery)it.next(); 
				 
				new Thread( new Runnable() { 
					public void run(){ 
						TConnection pooledConnection = null; 
						TLocalTransaction pooledtloc = null; 
						try { 
						pooledConnection = poolManager.getConnection(POOL_NAME); 
						accessor = pooledConnection.newXMLObjectAccessor( TAccessLocation.newInstance( collection ) , dom4jObjectModel ); 
						 
						TResponse response = accessor.xquery(query); 
						 
						TXMLObject o = response.getFirstXMLObject(); 
						StringWriter sw = new StringWriter(); 
						o.writeTo(sw); 
						} catch (TXQueryException tqe) { 
							log.debug(tqe.getMessage()); 
						} catch (TStreamWriteException tswe) { 
							log.debug(tswe.getMessage()); 
						} catch (TConnectionNotAvailableException tcnae){ 
							log.debug(tcnae.getMessage()); 
						} 
						finally { 
							long duration = System.currentTimeMillis()-start; 
							log.debug("update duration : " + duration + " ms"); 
							try { 
								if (pooledConnection!=null){ 
								pooledConnection.close(); 
								} 
							} 
							catch (TConnectionCloseException tcce){ 
								log.debug(tcce.getMessage());						 
							} 
						} 
					} 
				}).start(); 
			} 
	}

Thanks in advance for any piece of advice !
Thomas

ps :

2

Hi thomas porquet,

There are many factors which could impact things:

  • there is a thread created per query, so if there are more then 100 queries then these will be blocked waiting for a connection.
  • running lots of threads means more demand on CPU, memory etc and there comes a point where you can have too many threads and the system may spend time managing threads and memory as opposed to allowing the threads to do their job. Prehaps recode and reduce the number of threads.
  • is the java program running on the same machine as Tamino? If so one could impact the other especially if one of processes becomes CPU or memory intensive.
  • is the webserver becoming blocked with too many http requests? Is that sufficiently configured?
  • is Tamino sufficiently tuned to handle the number of concurrent connections?
  • check CPU / memory utilisation to see if there is anything obvious that you can start to look at.

The program looks ok in terms of utilising the connection pool manager. There is plenty than can be looked at but I think this may end up out of scope of this forum and I would therefore suggest speaking with your local support centre who may be able to provide further advice or possibly consultancy.