EventDataStore AND DES Integration

Eventdatastore and DES integration

NOTE:Below steps/data are based on SAG10.1 release

This document explains

  • Configuration steps to persist DES events into EventDataStore/Elasticsearch
  • How to query persisted DES events using DEPJDBC driver
  • Quick troubleshooting points on DEPJDBC driver

1. Event Data Store/ElasticSearch2.3.2 Configurations in Command Central to persist DES events

ElasticSearch2.3.2 Configurations

Configuring external ElasticSearch2.3.2

  1. When you unzip elasticsearch(https://www.elastic.co/downloads/past-releases/elasticsearch-2-3-2) the config\elasticsearch.yml file contains defaults. By default the cluster gets a random name. Edit the elasticsearch.yml file and name your cluster to your liking as below

      # Use a descriptive name for your cluster:

        cluster.name: SAG_EventDataStore

  1. You might also want to make sure that it binds to any address or it may default to localhost which will make it unreachable .

       # Set the bind address to a specific IP (IPv4 or IPv6):

        network.host: 0.0.0.0

        # Set a custom port for HTTP:

         http.port: 9200

         transport.tcp.port: 9300

  1. Start elasticsearch2.3.2 from bin\elasticsearch.bat and verify with http://<hostname>:<elasticsearch http.port>/_search(ex:http://localhost:9240/_search)

Persisting DES events from Apama to Event Data Store/ElasticSearch2.3.2

Create Apama Correlator instance in CCE

  • Login to CCE
  • Go to Installations >Local > Instances
  • Click + (Add Instance) button
  • Select 'Apama Correlator'
  • Enter 'Instance name' and 'Instance port'
  • Click Next >Finish
  • Open the instance created and go to Configuration tab
  • Select 'Arguments' in the dropdown
  • Click Edit button and enter 'Extra args' pointing to apama projects DigitalEventServices folder.
    ex:–config C:\Users\prch\workspace101\DESPubSub\config\connectivity\DigitalEventServices
  • Click Apply button
  • Select 'Initialization' in the dropdown
  • Click Edit button and enter apama projects .deploy file path
    ex:C:\Users\prch\workspace101\DESPubSub\config\launch\DESPubSub.deploy

Setting Apama Correlator instance to persist events to EventDataStore

  • Go to Digital Event Service tab
  • Go to Configuration tab
  • Select 'Event Persistence' in the dropdown
  • Click + button and select 'ElasticSearch 2.3.2'
  • Enter the 'service name' ,cluster url(elasticsearch url) ,clutser name,index(databasename) and click Save button
    Note:Cluster URL should point to elasticsearch tcpport and cluster name can be found from <installdir>\EventDataStore\data (default clutser name:SAG_EventDataStore)
  • Go to Service groups,click Default > Edit >+ button
  • Select the elsticsearch service with  'usage=Destination only'(Ex:ELS which is created above) and  save.

Start Apama Correlator instance in CCE

  • Go to Apama Project > Overview tab of the correlator
  • Click on 'Stop' image and Click Start
  • Click Ok button
  • Go to logs and verify no error in starting the correlator

Inject Apama Event and verify in ElasticSearch EventDataStore

  • In cmd prompt,go to <installdir>\Apama\bin
  • Execute engine_send command to inject the event file into correlator
    Ex:engine_send.exe -p 15903 C:\Users\prch\workspace101\DESPubSub\events\InjectEvent.evt
     
  • Wait for 30 second and then verify whether elasticsearch url shows the event which is persisted .

Persisting DES events from IS to Event Data Store/ElasticSearch2.3.2

Setting IS to persist DES events to EventDataStore

  • Login to CCE and click IS_default
  • Go to Digital Event Service tab
  • Go to Configuration tab
  • Select 'Event Persistence' in the dropdown
  • Click + button and select 'ElasticSearch 2.3.2'
  • Enter the 'service name' ,cluster url ,clutser name,index and click Save button
  • Go to Service groups,click Default > Edit >+ button
  • Select the elsticsearch service with  'usage=Destination only'(Ex:ELS which is created above) and  save.

Publish DES events

  • With the above setup,if we execute pub.publish:publish flow service then DES events gets persisted to elasticsearch/eventdatastore
  • Wait for 30 second and then verify whether elasticsearch url shows the event which is persisted.

2. Querrying DES events from Event Data Store/ElasticSearch2.3.2 using DEP JDBC driver

Prerequisites

  • Install ‘Digital Event Persisitence Driver 10.1 forJDBC’ node
  • Copy all the jars from ‘<installdir>\common\EventPersistence\JDBC’ to ‘<installdir>\IntegrationServer\instances\default\packages\WmJDBCAdapter\code\jars. This will allow JDBC adapter in IS to query EventDataStore using DEPJDBC driver.

Querrying EventDataStore from Apama

  • Use following properties in Apama to query EventDataStore using DEPJDBC driver
  1. JDBC Driver implementaion class:com.softwareag.evp.jdbc.EvpDriver
  2. JDBC URL: jdbc:evp:elasticsearch:2.3.2://<elasticSearchHost>:<HTTPport>/<ClusterName>/<indexName>?installDir=<path to JDBC driver directory>
  • Use JDBC Adapter in apama project and set the properties as below 
  1. Database type:Other
  2. Driver:com.softwareag.evp.jdbc.EvpDriver
  3. Driver classpath:C:\SoftwareAg\common\EventPersistence\JDBC\evp-jdbc-driver.jar
  • Add apama monitor and use com.apama.database.DBUtil event to connect to EventDataStore and query as shown in attached JdbcMonitor.mon file
  • Once connected to eventdatastore,data is fecthed and displayed in the log.

 

Querrying EventDataStore from InetgrationServer

  • Make sure you have copied required jars as mentioned in Prerequisites
  • Use following properties in IS to query EventDataStore using DEPJDBC driver.
  1. JDBC Datasource implementaion class:com.softwareag.evp.jdbc.EvpDataSource
  2. JDBC URL: jdbc:evp:elasticsearch:2.3.2://localhost:9340/SAG_EventDataStore/myindex
  3. installDir=<path to JDBC driver directory>
  • Create Connection in Integration Server > Adapters >webMethods Adapter for JDBC with above details .Specify JDBCURL &installdir in 'Other properties' while creating connection in IntegrationServer in the format "installdir=<path to JDBCdriverDirectory>;databaseURL= jdbc:evp:elasticsearch:2.3.2://<elasticsearchHost>:<elasticsearchHTTPport>/<ClusterName>/<indexName>"
  • Enable the connection
  • Create Adapter Service with SelectSQL template pointing to above connection
  • Choose proper ‘Table name’ and save
  • Execute the flow service and verify the data

Querrying EventDataStore from MashzoneNextGen

  • Use following properties in MashzoneNextGen to query EventDataStore using DEPJDBC driver 
  1. JDBC Driver implementaion class:com.softwareag.evp.jdbc.EvpDriver
  2. JDBC URL: jdbc:evp:elasticsearch:2.3.2://<elasticSearchHost>:<HTTPport>/<ClusterName>/<indexName>?installDir=<path to JDBC driver directory>
  • Go to MashzoneNextGen Admin console page
  • Add JDBC driver 
  • Add data source 
  • Create Dashboard with JDBC data source and see if data is fetched 

 

3. Troubleshooting hints

  • While querrying eventdatastore,make sure that jdbcurl is correctly pointing to host,port,clustername and indexname where eventdatastore is running.
  • If you are seeing an error “The driver or data source installation directory is a required property but has not been set” then make sure to pass installdir=<JDBCdriverPath> as a property in case of IS or as part of JDBCURL in case of Apama & MashzoneNextgen
  • If you are seeing an error “The specified feature is not supported by this driver or data source: UPDATE STATEMENT” then make sure that you are executing only SELECT query.
  • If you are seeing an error “ArrayIndexOutOfBoundsException” while enabling the JDBC connection in IS then make sure that JDBCURL contains indexname.
  • If you are seeing an error “No corresponding index exists for the specified catalog” then make sure that JDBCURL contains proper indexname.
  • If you are seeing an error “No parameters found in JDBC URL query string” in apama, then make sure that JDBCURL contains installdir property.

 

JdbcMonitor.mon (1.2 KB)