Hi All,
I am trying to work out how to consume a Kinesis stream using WM.io.
I have been able to connect to the stream, but I’m not sure how to go about consuming events off the stream.
Has anyone done this ?
I want to source events from the steam and then send them as messages to downstream systems.
Hi @can you share some light …on how you connected Kinesis Stream
So far i have a Flow service that does a:
Describe to get the shard details.
GetShardIterator to get the iterator.
GetRecords to get data.
However what I want is this to work like a trigger, constantly getting data from the stream, rather than just a few records …
Can you share the screen shot your connection configuration ? did you connect from IS admin page or a built a flow service to call Kinessis stream API ? or did you connect through JMS ?
I am using the webmethods.io Kinesis connector.
Option 1:we can use KCL option but we need to install kcl libraries. To support for languages other than Java is provided using a multi-language interface called the MultiLangDaemon .
Sample KCL code GitHub - awslabs/amazon-kinesis-client-nodejs: Amazon Kinesis Client Library for Node.js
It will also create Lease Table at DynamoDB table for creating checkpoint of processed records
Option 2:Use Firehose and configure Custom Https .It will cost extra for Firehose License.Also duplicate record will come to consumer(Not sure about this as not tested much) .
Option 3 : Use custom node js code for * [SubscribeToShard] that will use the HTTP/2 connection for up to 5 minutes and deliver all events by enhanced fan-out using the Kinesis Data Streams API .Use [ContinuationSequenceNumber] for next call or creating checkpoint at consumer end
option4: By Using GetRecords as already mentioned by you