working on compute infra at LinkedIn. ex-twitter/googlecloud/azure dist sys enthusiast github.com/ahmetbhttps://bsky.app/profile/ahmet.devhttps://bsky.app/profile/ahmet.dev/post/3mees6ywehk2m@ericchiang.bsky.social hey question about your synctest blog: do you really need that "+ time.Millisecond" or is there any determinism that the goroutine will get to run 3 times without it? https://oblique.security/blog/go-synctest/08 Feb 2026 20:40 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3mees6ywehk2mhttps://bsky.app/profile/ahmet.dev/post/3m7melz25g225@aphyr.woof.group.ap.brid.gy you thought LLM misquoting you was bad, no longer need to wait until LLM writes a jepsen article. https://sw.vtom.net/hn35/pages/90099678.html (from sw.vtom.net/hn35/news.html)10 Dec 2025 05:52 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3m7melz25g225https://bsky.app/profile/ahmet.dev/post/3m73wqqz2522eTurbopuffer (vector DB) architecture is a fairly interesting example of object-storage backed tiered DBs combined with data structure practices from DBs. https://turbopuffer.com/docs/architecture Here are the tradeoffs that makes it work: https://567-labs.github.io/systematically-improving-rag/talks/turbopuffer-engine/03 Dec 2025 17:02 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3m73wqqz2522ehttps://bsky.app/profile/ahmet.dev/post/3m6gdju6ass2aFull house from the KubeCon talk in Atlanta (500 people, holy sh*t)25 Nov 2025 02:52 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3m6gdju6ass2ahttps://bsky.app/profile/ahmet.dev/post/3m56cnvnnac2s#kubecon: roughly 14% flights into atlanta cancelled today, even more predicted for sunday/monday. https://www.flightaware.com/live/cancelled/today09 Nov 2025 04:50 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3m56cnvnnac2shttps://bsky.app/profile/ahmet.dev/post/3m56cduwyws27#kubecon talk finally coming together see you all there Thursday 11:45am or in the hallway https://kccncna2025.sched.com/event/27Fdd/evicted-all-the-ways-kubernetes-kills-your-pods-and-how-to-avoid-them-ahmet-alp-balkan-linkedin09 Nov 2025 04:44 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3m56cduwyws27https://bsky.app/profile/ahmet.dev/post/3lzplvpxmyk2jKubeCon goers: what are the Atlanta hotels you're booking that's not next to the boring downtown/convention center area?26 Sep 2025 04:10 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lzplvpxmyk2jhttps://bsky.app/profile/ahmet.dev/post/3lvobqfgno22zAnyone have recommended readings on any scalable k8s multi-cluster work API implementations? Planning to roll out a similar model on our own and I want to study the state of the art and the rationale behind the existing model.05 Aug 2025 17:55 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lvobqfgno22zhttps://bsky.app/profile/ahmet.dev/post/3lveovn43xs23#KubeCon schedule is out! Team LinkedIn will give 3 talks this year (myself included): 1️⃣ Making Application Rollouts Observable (lnkd.in/gmnM-8Z4) 2️⃣ How We Upgrade the Whole ML Training Fleet Bi-weekly(lnkd.in/gG_a_Etc) 3️⃣ Evicted! All the Ways Kubernetes Kills Your Pods (lnkd.in/ggm9HKZF)01 Aug 2025 22:24 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lveovn43xs23https://bsky.app/profile/ahmet.dev/post/3lv5oyasy6225Our team at LinkedIn will give 3 talks at the #KubeCon 2025! Looks like we'll be meeting in Atlanta again. 👋🏼 Stay tuned.30 Jul 2025 03:36 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lv5oyasy6225https://bsky.app/profile/ahmet.dev/post/3lv4xqravo22sDoes anyone know why all cloud-native open source ecosystem uses pull model to expose their /metrics (instead of pushing the metrics)? Pull model is really poor when the metric cardinality exposed on a single endpoint increases.29 Jul 2025 20:41 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lv4xqravo22shttps://bsky.app/profile/ahmet.dev/post/3ltkob3kwwc2yAs we test Kubernetes clusters beyond the supported 5,000+ nodes limit at LinkedIn, we're finding out 'List API' calls are a bottleneck in cluster reliability. So I'm publishing a deep dive about why this is a problem and what you can do about them. https://ahmet.im/blog/kubernetes-list-performance/09 Jul 2025 20:38 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3ltkob3kwwc2yhttps://bsky.app/profile/ahmet.dev/post/3lsckhzkckk2eOpen sourcing a new kubectl plugin called `kubectl node-resource` that queries allocations/utilizations for all/subset of the Kubernetes nodes. It has optimizations for large clusters. We've needed this in our set up to quickly answer questions. https://github.com/ahmetb/kubectl-node_resource/23 Jun 2025 21:43 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lsckhzkckk2ehttps://bsky.app/profile/ahmet.dev/post/3lsbzbjypcs2oGlad to have @madhavjivrajani.bsky.social join us at LinkedIn compute team this summer. I’m constantly impressed by the depth of his kube-apiserver expertise.23 Jun 2025 16:36 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lsbzbjypcs2ohttps://bsky.app/profile/ahmet.dev/post/3lqdk6ogdys2eDAYS SINCE ETCD BACKUPS SAVED OUR ASS IN PRODUCTION: 0 If you manage your Kubernetes clusters and think you won’t lose your etcd, you’re almost certainly wrong. It takes years for the disaster to happen and you’ll be glad you have invested in building a backup/restore system.29 May 2025 20:21 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lqdk6ogdys2ehttps://bsky.app/profile/ahmet.dev/post/3lovyw2wh6k2x@kelseyhightower.com hi Kelsey, what was your toto model again? can’t find after the twitter nuke ☹️11 May 2025 17:42 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lovyw2wh6k2xhttps://bsky.app/profile/ahmet.dev/post/3lly4jrt4fs2yFinally delivered our talk at #KubeCon to 500 attendees! Slides are up at kccnceu2025.sched.com/event/1txGQ/ and recording will be coming soon. If you wanna chat, I'm at the conference today.04 Apr 2025 09:36 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lly4jrt4fs2yhttps://bsky.app/profile/ahmet.dev/post/3llnw73xllk23Here we go again, this time in London. #kubecon maintainer summit31 Mar 2025 08:16 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3llnw73xllk23https://bsky.app/profile/ahmet.dev/post/3llfhqpayf22b🇬🇧 Looking forward to seeing y'all at our #KubeCon talk about how we're building a bare-metal compute platform at LinkedIn to run thousands of services on high-6-digit number of machines! Schedule: sched.co/1txGQ27 Mar 2025 23:36 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3llfhqpayf22bhttps://bsky.app/profile/ahmet.dev/post/3lld3prqnbc26Looking forward to seeing y'all at #KubeCon London next week! (Who's going?)27 Mar 2025 00:56 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lld3prqnbc26https://bsky.app/profile/ahmet.dev/post/3lj6qey6xnk2qNew article on my blog that explains all the various ways Kubernetes can evict your workloads. This wasn't all documented in one place so I'm hoping this helps the teams operating with Kubernetes in production. You'll also find a cheatsheet in the article. https://ahmet.im/blog/kubernetes-evictions/27 Feb 2025 20:32 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lj6qey6xnk2qhttps://bsky.app/profile/ahmet.dev/post/3lhp4cwtxd22eI might _attempt_ to write a blog about all the different things that causes Kubernetes evicts pods. There are so many, and every single one works differently.08 Feb 2025 21:58 +0000at://did:plc:6n3khpmbhb42dwtnf5sjfapg/app.bsky.feed.post/3lhp4cwtxd22e