Reading AWS Kinesis Data Stream messages from the console

Suppose we want to read AWS Kinesis Data Stream messages from the console. We can make it happen, as long as we have AWS CLI installed and configured.

Get main stream information:

aws kinesis describe-stream-summary --stream-name my-kinesis-stream-name

Get stream shard list:

aws kinesis list-shards --stream-name my-kinesis-stream-name

For intance, we want to read shardId-000000000000. We should get shard iterator for the beginning of the stream for this shard. Remember to insert shard id from the command above:

aws kinesis get-shard-iterator \
  --stream-name my-kinesis-stream-name \
  --shard-id shardId-000000000000 \
  --shard-iterator-type TRIM_HORIZON

Also, we can get shard iterator for the specific point in time (UNIX timestamp):

aws kinesis get-shard-iterator \
    --stream-name my-kinesis-stream-name  \
    --shard-id shardId-000000000000 \
    --shard-iterator-type AT_TIMESTAMP \
    --timestamp 1711582259

Set $iterator equals to iterator from the command above:

$iterator=<123456>

Run command:

aws kinesis get-records --shard-iterator $iterator

Or use this loop:

# Copy NextShardIterator from the output and set $iterator to the new value
# Repeat again.
# Another option, should have jq in order for it to work
# Try before actual usage!
while true; do
  output=$(aws kinesis get-records --shard-iterator $iterator)
  echo "Processing data..."
  iterator=$(echo $output | jq -r '.NextShardIterator')
  [[ -z "$iterator" ]] && break
done

Keep in mind, that you should use awslocal instead of aws, when working with localstack.

Telegram channel

If you still have any questions, feel free to ask me in the comments under this article or write me at promark33@gmail.com.

If I saved your day, you can support me 🤝

Leave a Reply

Your email address will not be published. Required fields are marked *