Helix Media Log

Receives media notifications from Helix MediaBus and logs them to S3

Status

Overview

The Helix Media Log service collects, processes, and stores media activity notifications for Adobe Helix projects. It tracks whenever media items (images, videos, documents, etc.) are added, updated, or deleted in the MediaBus system.

Key Features:

Automatic Batching: Groups media events by content bus ID for efficient processing
Scalable Architecture: Uses AWS Lambda, SQS, and S3 for automatic scaling
Compressed Storage: Stores logs as gzipped JSON files in S3
Large Message Support: Automatically handles messages exceeding SQS limits

For detailed architecture and implementation details, see PROJECT_OVERVIEW.md.

Architecture

MediaBus → SQS Input Queue → Lambda (Batch) → SQS FIFO Queue → Lambda (Write) → S3
                    ↑                                                            ↓
              EventBridge (1min)                                    helix-media-logs/

Installation

After deploying the service, you need to set up the infrastructure:

1. Create EventBridge Rule

aws events put-rule \
  --name media-log-every-minute \
  --schedule-expression "rate(1 minute)"

2. Grant Lambda Permissions

AWS_REGION=us-east-1
AWS_ACCOUNT_ID=118435662149

aws lambda add-permission \
  --statement-id "rule-media-log-every-minute" \
  --function-name "arn:aws:lambda:${AWS_REGION}:${AWS_ACCOUNT_ID}:function:helix3--media-log:v1" \
  --action 'lambda:InvokeFunction' \
  --principal events.amazonaws.com \
  --source-arn "arn:aws:events:${AWS_REGION}:${AWS_ACCOUNT_ID}:rule/media-log-every-minute"

3. Add Lambda as EventBridge Target

aws events put-targets \
  --rule media-log-every-minute \
  --targets "Id"="1","Arn"="arn:aws:lambda:${AWS_REGION}:${AWS_ACCOUNT_ID}:function:helix3--media-log:v1"

4. Create SQS Queues

# Input queue (standard)
aws sqs create-queue --queue-name helix-media-log

# Output queue (FIFO)
aws sqs create-queue \
  --queue-name helix-media-log.fifo \
  --attributes FifoQueue=true,ContentBasedDeduplication=false

5. Configure SQS Trigger

aws lambda create-event-source-mapping \
  --function-name helix3--media-log:v1 \
  --batch-size 1 \
  --event-source-arn arn:aws:sqs:${AWS_REGION}:${AWS_ACCOUNT_ID}:helix-media-log.fifo

Usage

Publishing Media Events

Services should publish media events to the helix-media-log SQS queue:

{
  "contentBusId": "355d601dd9b577248658b2b9ec3a9d7ddbc57b4428da7b8e532ab5aed6f",
  "timestamp": 1722427281000,
  "operation": "ingest",
  "mediaHash": "13872adbc8f226c65c00a81078b84ab4152476fc7",
  "mimeType": "image/png",
  "user": "[email protected]",
  "path": "/docs/faq",
  "originalFilename": "hero-image.png",
  "source": "gdoc-preview"
}

Media Event Fields

contentBusId (required): Unique identifier for the content bus (used for grouping and log organization)
timestamp (required): Unix timestamp in milliseconds when the event occurred
operation (required): Type of operation (ingest, reuse, delete, etc.)
mediaHash (required): Unique hash identifier for the media item
mimeType (required): MIME type of the media (e.g., image/png, image/jpeg)
user (required): Email or identifier of the user who performed the action
path (required): Path where the media is stored or referenced
originalFilename (optional): Original filename of the media
source (optional): Source system that generated the event (e.g., gdoc-preview, onedrive)

Accessing Logs

Logs are stored in S3 at s3://helix-media-logs/ organized by content bus ID:

helix-media-logs/
├── 355d601dd9b577248658b2b9ec3a9d7ddbc57b4428da7b8e532ab5aed6f/
│   ├── .index
│   ├── 2024-01-15-10-30-45-ABC123.gz
│   └── 2024-01-15-12-00-00-DEF456.gz
├── 455d601dd9b577248658b2b9ec3a9d7ddbc57b4428da7b8e532ab5aed6f/
│   └── ...
└── ...

Reading Logs:

# Download and decompress
aws s3 cp s3://helix-media-logs/355d601dd9b577248658b2b9ec3a9d7ddbc57b4428da7b8e532ab5aed6f/2024-01-15-10-30-45-ABC123.gz - | gunzip | jq .

# List all logs for a content bus
aws s3 ls s3://helix-media-logs/355d601dd9b577248658b2b9ec3a9d7ddbc57b4428da7b8e532ab5aed6f/

Development

Local Development

Start the development server:

npm install
npm run start

The server will be available at http://localhost:3000.

Testing Locally

curl -X POST localhost:3000 \
  -H 'Content-Type: application/json' \
  -d '[{
    "messageId": "msg-0",
    "body": "{\"contentBusId\":\"355d601dd9b577248658b2b9ec3a9d7ddbc57b4428da7b8e532ab5aed6f\",\"updates\":[{\"timestamp\":1722427281000,\"operation\":\"ingest\",\"mediaHash\":\"13872adbc8f226c65c00a81078b84ab4152476fc7\",\"mimeType\":\"image/png\",\"user\":\"[email protected]\",\"path\":\"/test.jpg\",\"originalFilename\":\"test.png\",\"source\":\"gdoc-preview\"}]}"
  }]'

Running Tests

npm test                    # Run unit tests
npm run test-postdeploy     # Run post-deployment tests
npm run lint                # Run linter

Deploying

npm run build               # Build and validate
npm run deploy              # Deploy to AWS

Configuration

Environment Variables

HLX_DEV_SERVER_HOST: Set when running locally

Lambda Configuration

Runtime: Node.js 22
Memory: 1024 MB
Timeout: 5 minutes
IAM Role: helix-service-role-s3-rw

Monitoring

Health Check

curl https://helix-pages.anywhere.run/helix3/media-log@v1/_status_check/healthcheck.json

CloudWatch Metrics

Monitor these metrics in AWS CloudWatch:

Lambda invocations and errors
SQS message counts (sent, received, deleted)
SQS age of oldest message
S3 bucket size and object count

Logging

All logs are available in CloudWatch Logs under the Lambda function log groups.

Troubleshooting

Messages Not Being Processed

Check SQS queue depth: aws sqs get-queue-attributes --queue-url <url>
Verify EventBridge rule is enabled: aws events describe-rule --name media-log-every-minute
Check Lambda errors in CloudWatch Logs

S3 Writes Failing

Verify IAM role has S3 write permissions
Check S3 bucket exists: aws s3 ls s3://helix-media-logs/
Review Lambda logs for error details

Performance Issues

Ensure SQS trigger batch size is set to 1
Monitor Lambda duration and memory usage
Check for visibility timeout issues in SQS

Documentation

PROJECT_OVERVIEW.md - Complete technical documentation
API.md - API documentation

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for details.

License

Apache-2.0. See LICENSE.txt for details.

Maintained by: Adobe Helix Team
Repository: https://github.com/adobe/helix-media-log

README