Using Apache Kafka for Grails event streams

Apache Kafka gives a Grails application a durable publish-subscribe channel for domain events. Instead of coupling an order controller to email, loyalty, analytics or fulfilment code, the controller publishes an “OrderPlaced” fact and independent consumers decide what to do with it. This event-driven style keeps request paths short, makes integrations easier to test and provides a replayable history when business data needs reconciliation.

Kafka stores records in partitioned topics, while consumer groups track their reading positions through offsets. A producer can publish thousands of events per second, and consumers can scale horizontally because each partition is assigned to one consumer within a group. Grails provides the domain model, controllers and services, while Kafka handles communication with other applications and processing systems.

The combination suits Australia’s distributed commercial landscape. Retail banks, insurers, superannuation platforms and logistics operators regularly connect systems that cannot share databases. Mobile banking, parcel tracking and transport updates also make stale data immediately visible to users. Teams working across Sydney, Melbourne and Brisbane must account for local deployment options, daylight-saving changes and Australian privacy obligations.

Map the first event stream

Start with one business capability rather than trying to turn the whole application into an event bus. A useful first topic might represent completed orders, captured payments or updated delivery status. Identify the service that owns the fact, the applications that need to react and the business outcome each consumer provides. This keeps topic boundaries understandable and makes failures easier to diagnose.

Event contracts should capture what happened without exposing the producer’s internal database model. The same source-first discipline appears in a guide to photographing cathedral interiors, where the photographer observes available light rather than manufacturing conditions that were not present. Likewise, a domain event records an actual state change and its relevant context.

A practical event contract contains:

Keep personal information out of an event unless a consumer genuinely requires it. The partition key should normally identify the business entity, such as an order or customer identifier. Kafka then preserves ordering for that entity within a partition without forcing unrelated records into the same queue. Store timestamps in UTC and convert them for display, since Sydney and Melbourne change their UTC offset during daylight saving while Brisbane remains on Australian Eastern Standard Time.

Configure Kafka without framework friction

Grails applications already use Spring-managed components, so Spring Kafka provides a natural integration path. Add a Spring Kafka version compatible with the Grails release and keep the broker address outside source control. Local development can use an environment variable with a localhost default, while deployed applications receive their configuration from the platform.

dependencies {
    implementation 'org.springframework.kafka:spring-kafka'
}

Use JSON serializers for straightforward event transport, provided the application restricts the packages it is willing to deserialize. Package names, trusted-package settings and broker endpoints differ between projects, so the example values should be replaced rather than copied unchanged.

spring:
  kafka:
    bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092}
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
    consumer:
      group-id: grails-example
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: com.example.events

A separate consumer group receives its own copy of every record. This allows a loyalty service, a fulfilment worker and an analytics application to evolve independently. The site's guidance on custom auto-configuration for plugins is useful when the same producers, consumers and configuration settings need to ship across several Grails applications.

Publish durable domain events

A publisher should translate a completed business operation into a stable event and send it to the appropriate topic. The event class belongs in the domain layer, while the Kafka-specific publisher belongs in an integration service. This separation prevents transport concerns from spreading through controllers and domain services.

class OrderPlacedEvent {
    String eventId
    String orderId
    Instant occurredAt
    String customerId
}

@Component
class OrderEventPublisher {
    KafkaTemplate<String, OrderPlacedEvent> kafkaTemplate

    void publish(OrderPlacedEvent event) {
        kafkaTemplate.send('orders.placed', event.orderId, event)
    }
}

The event identifier supports tracing and duplicate detection, while the order identifier provides a meaningful partition key. Production code should also check the result of asynchronous sends and use an idempotent producer where repeated requests could otherwise create duplicate records.

When several Grails applications publish the same contract, a shared domain logic plugin can keep the event class and validation rules in one place. A direct database write followed by a Kafka send is not atomic: the application could update the order and crash before the event reaches the broker. For critical operations, write the business change and an outbox record in one database transaction, then publish the outbox entry asynchronously.

Consume events with idempotent handlers

Spring Kafka’s @KafkaListener annotation connects a consumer method to a topic and consumer group. Consumers within a group divide partitions between themselves, but every group maintains its own offsets. This lets the Grails application resume processing after a restart or replay records by resetting a group’s position.

A consumer should assume that the same event can arrive more than once. Network retries, a rebalance or a process restart after processing but before committing the offset can all produce duplicates. Robust handlers follow a small set of delivery rules:

An idempotency table can contain the event identifier alongside the resulting business reference. When a repeated event arrives, the handler detects the identifier and exits successfully, allowing Kafka to advance. Where the local work belongs in a database transaction, the offset still cannot share that transaction with Kafka; duplicate protection therefore remains essential.

This matters in Australian customer journeys such as parcel tracking or mobile banking. A duplicated notification is inconvenient, while a duplicated payment or loyalty credit is financially damaging. Consumer groups should also use bounded concurrency so that a slow database or external API creates visible lag rather than exhausting the Grails thread pool. Local Kafka and embedded test servers are useful for verifying retries, duplicate delivery and consumer-group behaviour before deployment.

Secure and observe the event pipeline

Treat a Kafka cluster as a distributed database with an active network boundary. Use TLS for data in transit, SASL authentication such as SCRAM, and topic-level access controls that grant applications only the operations they require. Store broker credentials in the deployment platform’s secret manager rather than in application.yml, build logs or source control.

Australian privacy duties reach events as well as conventional databases. Personal information in Kafka may be covered by the Privacy Act 1988 and the Australian Privacy Principles, so minimise sensitive fields before publication, restrict retention and document how corrections or erasure requests affect retained records. Consumer offsets do not erase data. If events trigger commercial emails or SMS messages, the Spam Act 2003 and relevant consent rules may also apply.

Schema evolution prevents a new producer version from breaking existing consumers. JSON works well for simple contracts, while Avro or Protobuf provides stronger compatibility checks when many services share a topic. Register schemas in a schema registry, reject incompatible changes and retain older readers long enough for planned deployments. A schema registry does not replace application validation, but it makes accidental contract changes visible earlier.

Operational visibility should include consumer lag, production error rate, broker disk usage, under-replicated partitions and processing duration. Include the event identifier, topic and partition in structured logs, but exclude sensitive payloads. Alerts should indicate when lag continues growing or a consumer has entered a retry loop. Because Kafka retains history, these records also support reconciliation after an Australian payment provider, logistics operator or internal service falls temporarily out of step.

Deploy and prove the event stream

A managed Kafka service usually provides the fastest route for a Grails team because it handles broker replacement, storage expansion and some operational maintenance. Select an Australian region to reduce latency and support local data-residency requirements. Sydney and Melbourne are common regional choices, while Brisbane users should test the latency of the nearest available deployment rather than assuming every provider operates in their city.

Self-managed Kafka offers more control but adds responsibility for capacity, upgrades, monitoring and recovery. Either way, document replication settings, retention limits, partition counts, schema ownership and recovery procedures. Run consumer-group and broker-failure tests in staging, and schedule maintenance around Australian business hours while accounting for the daylight-saving transitions that affect Sydney and Melbourne.

Start with one non-production capability: create a topic, publish a versioned event, consume it through two different groups, restart a consumer to demonstrate replay, and verify that duplicate delivery does not repeat the business action. Record the partition key, retention period and ownership before expanding the stream. A working producer, replayable log and idempotent consumer will give the team a reliable foundation for later payment, fulfilment and customer-notification workflows.