E2E workflow: deploy the template into a throwaway AWS account and read telemetry back

[PetrHeinz]

A new E2E workflow deploys the template into a throwaway AWS account the way a customer sets it up and reads the resulting telemetry back from Better Stack. It runs on every pull request and push to main with the template from the checkout, and daily with the published S3 template, which must then equal main.

What a run does:

• checks that the docs article names every true/false template parameter, both outputs and this repository
• creates an aws source in the E2E testing team, warms it up until the ingesters accept its token, and pre-creates two log groups: one named after the run, and /e2e/static, which every run recreates under the same name
• deploys cloudformation/full/better-stack-full.yaml with that source's ClusterId, token and id, pastes the IntegrationRoleArn and ExternalId outputs back into the source, and waits until the sync has put the BetterStackLogs subscription filter on the run's log group
• checks that the sync leaves /e2e/static, aws/spans and the CloudTrail group unsubscribed, because Better Stack remembers them from earlier runs, then subscribes them through the API. This is current behaviour, and the run fails with a pointer to the internal ticket once it changes.
• writes 20 log lines into each log group, 3 datapoints of a custom metric, a two-segment X-Ray trace and a PutRetentionPolicy call, then reads all of them back through the team's SQL API connection with exact counts. It counts again a minute later to catch duplicate delivery and checks that the stored trace id matches the X-Ray one.
• deletes the source, the stack, the two buckets the stack retains by design, aws/spans, /aws/application-signals/data and the /e2e/* log groups, and fails if it cannot list or delete something

A run within an hour of the previous run's stack deletion checks the static group, aws/spans and CloudTrail leniently. It waits only for the run's own log group and the metric, and when one of those three delivered nothing, it adds a warning to the log and the job summary instead of failing. Any other wrong count still fails. The template names its IAM roles per account, so every run recreates them under the same names, and AWS keeps using sessions of the deleted roles for up to an hour:

• CloudWatch Logs holds the subscription role's session per log group name. Delivery with a deleted role's session fails with a non-retryable error, so the filter pauses for 10 minutes and skips whatever arrives meanwhile. It fails again on every retry until the session expires, even when the log group is deleted and recreated with a new filter. aws/spans, the CloudTrail group and /e2e/static have the same names in every run.
• CloudTrail holds its role's session for writing into /aws/cloudtrail/better-stack the same way, and writes nothing there until that session expires.

Runs in quick succession lose exactly this data: the run-named group always arrives, while the fixed-name groups fail at random. A run more than an hour after the previous one checks everything strictly. Customers keep their roles and are not affected. A customer who deletes and redeploys the stack would lose up to an hour of logs from the groups that were subscribed before.

When something does not arrive within 20 minutes, the run prints the incoming, forwarded and delivery-error counts of each subscribed log group.

The account is better-stack-e2e-aws-stack (251917137055), a member of the Better Stack organization that holds nothing else. AWS access is GitHub OIDC through the github-actions-e2e role of a one-time stack that is already deployed in the account, so no AWS keys live in GitHub. The role has administrator access because the template creates IAM roles, Lambdas, Firehose streams, CloudTrail and S3, and the account is otherwise empty. The template's role and bucket names are fixed per account, so a concurrency group keeps it to one deploy at a time. Pull requests from forks are skipped, since they get no OIDC token.

The job summary shows the counts, sample rows and the cleanup. Secrets TEST_TEAM_TELEMETRY_API, TEST_TEAM_CONNECTION_USERNAME and TEST_TEAM_CONNECTION_PASSWORD are set. The LINEAR_* secrets are set as well: a failure on main or on the schedule opens a High-priority issue in the Telemetry team's Plan state, in the Ingestion for everything project, assigned to Petr.

🤖 Generated with Claude Code