swarm

swarm

Go channels for distributed queueing and event-driven systems

sub-module	doc	features	about
			API & Kernel
			AWS EventBridge
			AWS DynamoDB Stream
			AWS S3 Event
			AWS SQS Events
			AWS SQS and SQS FIFO
			AWS WebSocket API
			AWS SNS
			AWS Kinesis Events
			AWS Kinesis
			AWS ElastiCache
			MQTT

From Chaos to Channels

"Finally, messaging that feels like Go!" — Distributed systems, done right.

Writing distributed, event-driven systems in Go today is harder than it should be - a lesson learned from a decade building such systems in Erlang. Vendor APIs are clunky, non-idiomatic, and tightly coupled to specific messaging brokers.

swarm makes asynchronous, distributed messaging in Go idiomatic, testable, and portable by expressing queueing/event-driven systems through Go channels instead of vendor-specific APIs.

User Guide | Playground | Getting started | Examples | Philosophy

Quick Start

The example below shows the simplest way to enqueue and dequeue messages using AWS SQS.

package main

import (
  "github.com/fogfish/swarm"
  "github.com/fogfish/swarm/broker/sqs"
  "github.com/fogfish/swarm/emit"
  "github.com/fogfish/swarm/listen"
)

// Example message
type Order struct {
  ID     string  `json:"id"`
  Amount float64 `json:"amount"`
}

func main() {
  // create broker for AWS SQS
  q := sqs.Endpoint().Build("aws-sqs-queue-name")

  // create Golang channels
  rcv, ack := listen.Typed[Order](q.Listener)
  out := swarm.LogDeadLetters(emit.Typed[Order](q.Emitter))

  // Send messages
  out <- Order{ID: "123", Amount: 100.0}

  // use Golang channels for I/O
  go func() {
    for order := range rcv {
      processOrder(order.Object)
      ack <- order  // acknowledge processing
    }
  }()

  q.Await()
}

func processOrder(order Order) {
  // Your business logic here
}

That's it! You're now using distributed messaging with native Go channels.

• Try this code in playground
• Continue to Getting Started for advanced configuration
• See more examples by browsing /broker/*/examples/
• Check the design pattern Distributed event-driven Golang channels for deep-dive into library philosophy.

Continue reading to understand the purpose of the library and why it exists.

What is swarm?

swarm is a Go library that solves the complexity of distributed, event-driven systems by abstracting external messaging queues (like AWS SQS, AWS EventBridge, RabbitMQ, Kafka, etc.) behind type-safe Go channels.

By aligning with Go's native concurrency model, swarm encourages developers to think in terms of message flows rather than request-response cycles, which reflects the asynchronous and unreliable reality of distributed systems. This mindset shift leads to more resilient, scalable, and maintainable architectures — where complexity is not hidden, but made explicit and manageable.

• Immediate Productivity: Use Go channel patterns you already know, no need to learn new APIs.
• Smooth Testing: Write unit tests with in-memory channels; switch to real systems for integration.
• Future-Proof: Swap messaging technologies without changing your business logic.
• Serverless-Ready: Designed for both long-running services and ephemeral serverless functions.
• Scale-Out: Idiomatic architecture to build distributed topologies and scale-out Golang applications.

Think of swarm as net.Conn for distributed messaging: a universal, idiomatic interface you can depend on, regardless of the underlying transport.

Below we discussed why swarm exists and what problems it solves.

Why Choose swarm?

Traditional messaging libraries have fundamental issues

swarm solves these problems by providing a universal, idiomatic interface built on Go's concurrency model — embracing the asynchronous nature of distributed systems, making message flows explicit, and hiding vendor-specific complexity behind familiar Go channels.

See practical scenarios for swarm in Storytelling.

Getting started

The library requires Go 1.24 or later due to usage of generic alias types.

The latest version of the library is available at main branch of this repository. All development, including new features and bug fixes, take place on the main branch using forking and pull requests as described in contribution guidelines. The stable version is available via Golang modules.

Use go get to retrieve the library and add it as dependency to your application.

go get -u github.com/fogfish/swarm

When to Use swarm

Perfect for:

• Event-driven architectures - Decouple services with async messaging
• Serverless applications - Zero-boilerplate event consumers on AWS Lambda
• Microservices - Replace fragile HTTP calls with resilient messaging
• High-throughput systems - Scale message processing horizontally
• Multi-cloud applications - Abstract away broker differences

You'll love it if you:

• Want type-safe messaging without vendor lock-in
• Need to unit test message-driven code easily
• Prefer Go channels over learning broker-specific APIs
• Value clean, testable architecture patterns

Advanced Usage and Next steps

• Learn why we built swarm.
• User guide help you with adopting the library.
• The library supports a variety of brokers out of the box. See the table at the beginning of this document for details. If you need to implement a broker that isn’t supported yet, refer to Bring Your Own Broker for guidance.

Real-World Success Stories

Here's how swarm solves actual production problems that teams face every day.

E-commerce Order Processing: The Cascading Failure Problem

Imagine you're building an e-commerce platform that processes thousands of orders per minute during Black Friday sales. Your order processing system needs to:

• Validate payment information
• Update inventory levels
• Send confirmation emails
• Update analytics systems
• Trigger fulfillment processes

What are problems with the traditional approach:

• Blocking Operations: Each service call blocks until completion, making the system slow during peak loads
• Cascading Failures: If the email service is down, the entire order process fails, even though the core business logic succeeded
• Poor Scalability: Cannot scale individual components independently

func ProcessOrder(order *Order) error {
  // Synchronous calls create cascading failures
  if err := paymentService.Charge(order.Payment); err != nil {
    return err // Customer sees error immediately
  }
    
  if err := inventoryService.Reserve(order.Items); err != nil {
    return err // Payment charged but inventory failed
  }
    
  if err := emailService.SendConfirmation(order); err != nil {
    return err // Everything fails if email is slow
  }
    
  return nil
}

Using the swarm library, we benefit from

• Immediate Response: Customers get instant order confirmation
• Fault Tolerance: If email service is down, orders still process successfully
• Independent Scaling: Each service can scale based on its own load patterns

func ProcessOrder(order *Order) error {
  // Immediate response to customer
  if err := validateOrder(order); err != nil {
    return err
  }

  // Async processing via channels
  orderEvents <- OrderCreated{Order: order}
    
  return nil // Customer gets immediate confirmation
}

// Background processing
go func() {
  for event := range orderEvents {
    paymentQueue <- PaymentRequest{Order: event.Order}
    inventoryQueue <- InventoryUpdate{Order: event.Order}
    emailQueue <- EmailNotification{Order: event.Order}
  }
}()

Serverless Event Processing: The SDK Complexity Problem

You're building a serverless application on AWS that needs to process events from multiple sources - SQS queues, S3 bucket notifications, EventBridge events, and DynamoDB streams. Each service has its own SDK with different patterns.

// Different patterns for each service
func handleSQSEvent(event events.SQSEvent) error {
  for _, record := range event.Records {
    // SQS-specific parsing and acknowledgment
    body := record.Body
    receiptHandle := record.ReceiptHandle
    // ... SQS-specific error handling
  }
}

func handleS3Event(event events.S3Event) error {
  for _, record := range event.Records {
    // S3-specific parsing
    bucket := record.S3.Bucket.Name
    key := record.S3.Object.Key
    // ... S3-specific error handling
  }
}

Using the swarm library, we benefit from

• Unified Interface: Same code patterns work across all AWS services
• Easy Testing: Mock implementations for local development
• Technology Agnostic: Can switch from SQS to EventBridge without code changes

events := eventbridge.Must(eventbridge.Listener().Build())
upload := s3.Must(s3.Listener().Build())
orders := sqs.Must(sqs.Listener().Build())

listen.Typed[UserEvent](events)
listen.Typed[DataUploaded](upload)
listen.Typed[Order](orders)

Microservices Communication: The Synchronous Trap

You have a microservices architecture where the user service, notification service, and analytics service need to communicate. Traditional REST APIs create tight coupling and cascading failures.

// Synchronous REST calls create brittleness
func CreateUser(user *User) error {
  // Save user
  if err := userDB.Save(user); err != nil {
    return err
  }

  // Synchronous calls to other services
  if err := notificationService.SendWelcomeEmail(user); err != nil {
    return err // User creation fails if email fails
  }

  if err := analyticsService.TrackUserCreated(user); err != nil {
    return err // User creation fails if analytics fails
  }

  return nil
}

Using the swarm library, we benefit from

• Resilient: Core functionality works even if downstream services are down
• Scalable: Services can process at their own pace
• Evolvable: Easy to add new services without changing existing code

// Async communication via channels
func CreateUser(user *User) error {
  // Core business logic
  if err := userDB.Save(user); err != nil {
      return err
  }

  // Async notifications
  userEvents <- UserCreated{User: user}

  return nil // Immediate success
}

// Services consume events independently
go func() {
  for event := range userEvents {
    notificationQueue <- WelcomeEmail{User: event.User}
    analyticsQueue <- UserCreatedEvent{User: event.User}
  }
}()

How To Contribute

The library is Apache Version 2.0 licensed and accepts contributions via GitHub pull requests:

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Added some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

The build and testing process requires Go version 1.24 or later.

build and test library.

git clone https://github.com/fogfish/swarm
cd swarm
go test ./...

commit message

The commit message helps us to write a good release note, speed-up review process. The message should address two question what changed and why. The project follows the template defined by chapter Contributing to a Project of Git book.

bugs

If you experience any issues with the library, please let us know via GitHub issues. We appreciate detailed and accurate reports that help us to identity and replicate the issue.

Community & Support

• Documentation: User Guide • API Reference
• Questions: GitHub Discussions for Q&A and ideas
• Issues: GitHub Issues for bug reports and feature requests
• Examples: Live examples for all supported brokers

License