AWS Hero Staff Eng @ Datadog Streaming at: twitch.tv/aj_stuyvenberg Videos at: youtube.com/@astuyve I write about serverless minutia at aaronstuyvenberg.com/ https://bsky.app/profile/ajs.bsky.socialhttps://bsky.app/profile/ajs.bsky.social/post/3meoa5hfmp22wThe clanker wars have begun: [contains quote post or other embedded content]12 Feb 2026 14:44 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3meoa5hfmp22whttps://bsky.app/profile/ajs.bsky.social/post/3lz4fxwnnw32nI use AWS a ton but Lambda still astounds me. Throw some code in a function, send 1m requests as fast as you can. It ate up all available file descriptors on my little t3 box and still ran 18k RPS with a p99 of 0.3479s. Not many services can go from 0 to 18k RPS instantaneously with this p99.18 Sep 2025 13:03 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lz4fxwnnw32nhttps://bsky.app/profile/ajs.bsky.social/post/3lvobjsfimb2zLambda now charges for init time, so it's useful to count sandboxes which are proactively initialized but never receive a request. Here's what happens after a 10k request burst. Hundreds of sandbox shutdowns, along with 22 sandboxes which were spun up but never received a request.05 Aug 2025 17:51 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lvobjsfimb2zhttps://bsky.app/profile/ajs.bsky.social/post/3lvdwcwuron2eHappy Lambda Init Billing day to those who celebrate. Fix your cold starts!01 Aug 2025 15:04 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lvdwcwuron2ehttps://bsky.app/profile/ajs.bsky.social/post/3lvchqsywai2mNEW: Lambda can now send up to 200mb payloads using response streaming! I assume this is mostly directed at LLM inference workloads, where chatbots can stream large amounts of data over the wire as it becomes available.01 Aug 2025 01:10 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lvchqsywai2mhttps://bsky.app/profile/ajs.bsky.social/post/3lv76btqkc52aI've long been an advocate for the Lambda Web Adapter project which lets anyone pretty easily ship an app to Lambda without learning about the event model/API. Honestly AWS should simply support this natively.30 Jul 2025 17:43 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lv76btqkc52ahttps://bsky.app/profile/ajs.bsky.social/post/3luzu75baor2g"We run benchmarks continually across all of our competitors, not just queries - even connections, ensuring we don't add any latency at all." @isamlambert Performance is such a competitive advantage which easily slips away if you're not constantly paying attention to it.28 Jul 2025 14:59 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3luzu75baor2ghttps://bsky.app/profile/ajs.bsky.social/post/3lukt6tfjpa2lHere's another 33% cold start reduction, which comes from deferring expensive decryption calls made to AWS Secrets Manager until the secret is actually needed. Lazy loading is great!22 Jul 2025 15:31 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lukt6tfjpa2lhttps://bsky.app/profile/ajs.bsky.social/post/3luapykfbk72kHere's how to visualize a 100% memory allocation improvement! A recent stress test revealed that malloc calls bottlenecked when sending > 100k spans through the API and aggregator pipelines in Lambda.18 Jul 2025 15:07 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3luapykfbk72khttps://bsky.app/profile/ajs.bsky.social/post/3lu6arbpojw2nNEW: A recent blog post went viral in the AWS ecosystem, about how there's a silent crash in AWS Lambda's NodeJS runtime. Today I'll step you through the actual Lambda runtime code which causes this confusing issue, and walk you through how to safely perform async work in Lambda:17 Jul 2025 15:29 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lu6arbpojw2nhttps://bsky.app/profile/ajs.bsky.social/post/3ltwrigp7ds2kLambda's fleet management shutdown algorithm is learning faster! I'm calling this function every 8 or so minutes. At first the gap from invocation to shutdown is about 5-6 minutes, which was the fastest I've observed during previous experiments.14 Jul 2025 16:07 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3ltwrigp7ds2khttps://bsky.app/profile/ajs.bsky.social/post/3ltp5m2noq425NEW: AWS is rolling out a new free tier beginning July 15th!! New accounts get $100 in credits to start and can earn $100 exploring AWS resources. You can now explore AWS without worrying about incurring a huge bill, this is great! https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/free-tier.html11 Jul 2025 15:23 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3ltp5m2noq425https://bsky.app/profile/ajs.bsky.social/post/3ltmnturyix2kYou should care about your p99! By improving the function cold start time, the service on the left performed: 2x faster in RPS and thus, duration. p99 from 1.52s -> .949s The code and functionality is identical, but improving the cold start from 816ms to 301ms made all the difference.10 Jul 2025 15:36 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3ltmnturyix2khttps://bsky.app/profile/ajs.bsky.social/post/3lthnz2ga6e2gQuick PSA to make sure you're using a DLQ and setting a max receive count for SQS, otherwise you may find yourself looking at a flamegraph like this. Hundreds of attempts, multiple messages in queue and not burning down and average age of message ticking up! Seems common knowledge, but...08 Jul 2025 15:55 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lthnz2ga6e2ghttps://bsky.app/profile/ajs.bsky.social/post/3lt2zi4vijw2oReal-time audio processing is an extremely interesting corner of performance engineering. You've gotta process data faster than it streams in, otherwise the work is wasted. The engineering team behind CoScreen just open sourced a new library, dtln-rs, suitable for removing noise from all kinds https://www.datadoghq.com/blog/engineering/noise-suppression-library/03 Jul 2025 15:16 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lt2zi4vijw2ohttps://bsky.app/profile/ajs.bsky.social/post/3lqd3d4ljpb2sMonitoring Lambda sandbox shutdowns reveals an interesting scaling behavior After a request spike, Lambda waits ~10m before reaping 2/3rds of sandboxes. 5m later it begins reaping the rest. Presumably this helps smooth latency during retry storms, or if traffic returns!29 May 2025 15:55 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lqd3d4ljpb2shttps://bsky.app/profile/ajs.bsky.social/post/3lpozrsjtp22sHere's another huge p99 cliff! One unreasonably effective way to lower the average latency of a service is to minimize the causes of p99 events. Here, we've managed to absolutely crush the Max Post Runtime Duration from ~80ms to 500µs!21 May 2025 16:34 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lpozrsjtp22shttps://bsky.app/profile/ajs.bsky.social/post/3lojamacjjg2fTruly incredible to learn that Epic Games isn't self hosted on a VPS.06 May 2025 15:55 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lojamacjjg2fhttps://bsky.app/profile/ajs.bsky.social/post/3lo53p7x2bc2ePRICE CUT: Lambda slashes the price of cloudwatch logs for high volume. From $.50/gb down to $0.05/gb after 50TB. You've gotta be spending a decent chunk of $$$ on cloudwatch logs for this to help, but still – a price cut is a price cut!01 May 2025 19:55 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lo53p7x2bc2ehttps://bsky.app/profile/ajs.bsky.social/post/3lo4q6k3bnb2iCombining profiler data with claude code feels super powerful. Just drop a pprof file into a project, explain the dimensions of the profile, then let the LLM make suggestions to solve the hot spots. Instant performance boost01 May 2025 16:29 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lo4q6k3bnb2ihttps://bsky.app/profile/ajs.bsky.social/post/3lny2hgkxd22dNEW: AWS Lambda will now begin billing for the INIT phase for all runtimes (not just custom/container runtimes). Your cold starts cost money now! My hottest take is that AWS should have done this years ago.29 Apr 2025 19:49 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lny2hgkxd22dhttps://bsky.app/profile/ajs.bsky.social/post/3lninquqtjc25The evolution of my program when each branch is benchmarked & profiled before merging23 Apr 2025 16:52 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lninquqtjc25https://bsky.app/profile/ajs.bsky.social/post/3lmhvwhzwrh2wCheck out that p99 cliff! I spent 2024 shipping our next-generation Lambda Extension, which offers an 82% improvement in cold start time, better aggregation and flushing options, and lower overall overhead – all in a substantially smaller binary package. More: https://www.datadoghq.com/blog/engineering/datadog-lambda-extension-rust/10 Apr 2025 16:21 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lmhvwhzwrh2whttps://bsky.app/profile/ajs.bsky.social/post/3lk22awkiis2eCloudflare isn't pulling any punches with this twitter outage. Absolutely brutal:10 Mar 2025 17:10 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lk22awkiis2ehttps://bsky.app/profile/ajs.bsky.social/post/3li6sz2c6jc2xAbsolutely incredible piece about a font you've seen everywhere: [contains quote post or other embedded content]15 Feb 2025 03:54 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3li6sz2c6jc2xhttps://bsky.app/profile/ajs.bsky.social/post/3lge3nq23ar2j🚨 NEW VIDEO: Good clocks could be huge for us devs – so now I've finally made a video about databases, how they are distributed, and why precise clocks can change things. This can also go horribly wrong, so we'll cover that too. Check it out – https://buff.ly/4hhkVtw22 Jan 2025 19:21 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3lge3nq23ar2jhttps://bsky.app/profile/ajs.bsky.social/post/3ldoakrbcyw2eAlthough the market is decidedly less frothy and VC money is harder than ever to raise, startups can still win – and win big. Resharing this story about how WarpStream reinvented Kafka and walked away with $220 in only 13 months. https://buff.ly/400AYpJ19 Dec 2024 16:00 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3ldoakrbcyw2ehttps://bsky.app/profile/ajs.bsky.social/post/3ldjhhuqxcb2lShould you use Lambda Function Warmers in 2025? Probably not! Here's some preliminary data. It's 200 requests at around 20 requests per second (2 runs). The warmer function attempts to create 10 concurrent sandboxes. Notice that the number of cold starts (~75) is the same!17 Dec 2024 18:20 +0000at://did:plc:fme35exbc57cqgwestca5nd7/app.bsky.feed.post/3ldjhhuqxcb2l