Training LLM's at huggingface | hf.co/sciencehttps://bsky.app/profile/eliebak.hf.cohttps://bsky.app/profile/eliebak.hf.co/post/3ld25uihfbc2mWOW, Gemini Flash 2.0 is really impressive. Wondering about the size of this supposedly smol model. One odd thing is that the model seems to lose some ability with long contexts compared to Flash 1.5. If any google friends could share insights, I'd love to hear them!11 Dec 2024 16:19 +0000at://did:plc:ms5fdduxysmzdoplpbtbcbo4/app.bsky.feed.post/3ld25uihfbc2mhttps://bsky.app/profile/eliebak.hf.co/post/3lchp3us25226Hey, I'll be at neurips next week! My DM are open if you want to meet and talk about pre-training/data/whatever you want 🫡04 Dec 2024 08:06 +0000at://did:plc:ms5fdduxysmzdoplpbtbcbo4/app.bsky.feed.post/3lchp3us25226https://bsky.app/profile/eliebak.hf.co/post/3lcfixovkck2dGoogle patent on "Training of large neural network". 😮 I don't know if this give much information but by going quickly through it seems that: - They are not only using "causal language modeling task" as a pre-training task but also "span corruption" and "prefix modeling". (ref [0805]-[0091])03 Dec 2024 11:11 +0000at://did:plc:ms5fdduxysmzdoplpbtbcbo4/app.bsky.feed.post/3lcfixovkck2dhttps://bsky.app/profile/eliebak.hf.co/post/3lc6ff25vsk2lWhat else should we log during LLM training? Right now, it's just loss, grad_norm, and evals, but I want to log more to have a better understanding of pre-training. Thinking about adding stuff like entropix metrics (agreement, varentropy?) Any thoughts or cool ideas?30 Nov 2024 15:19 +0000at://did:plc:ms5fdduxysmzdoplpbtbcbo4/app.bsky.feed.post/3lc6ff25vsk2lhttps://bsky.app/profile/eliebak.hf.co/post/3lbwmqu2sn22e10000% agree with omar, this is totally disproportionate [contains quote post or other embedded content]27 Nov 2024 13:09 +0000at://did:plc:ms5fdduxysmzdoplpbtbcbo4/app.bsky.feed.post/3lbwmqu2sn22ehttps://bsky.app/profile/eliebak.hf.co/post/3lbwdbrg2mk2pWe’re looking for an intern to join our SmolLM team! If you’re excited about training LLMs and building high-quality datasets, we’d love to hear from you. 🤗 US: https://apply.workable.com/huggingface/j/0643507FC5/ EMEA: https://apply.workable.com/huggingface/j/0643507FC5/27 Nov 2024 10:20 +0000at://did:plc:ms5fdduxysmzdoplpbtbcbo4/app.bsky.feed.post/3lbwdbrg2mk2phttps://bsky.app/profile/eliebak.hf.co/post/3lbuii763p22sThe SmolLM series has a new member: say hi to SmolVLM! 🤏 It uses a preliminary 16k context version of SmolLM2 to tackle long-context vision documents and higher-res images. And yes, we’re cooking up versions with bigger context lengths. 👨‍🍳 Try it yourself here: https://huggingface.co/spaces/HuggingFaceTB/SmolVLM26 Nov 2024 16:47 +0000at://did:plc:ms5fdduxysmzdoplpbtbcbo4/app.bsky.feed.post/3lbuii763p22shttps://bsky.app/profile/eliebak.hf.co/post/3l7tfpyrfdc2fHey babe, wake up, we just dropped a new SmolLM 🫡 Fully open-source. We’ll release a blog post soon to detail how we trained it. I'm also super excited about all the demos that will come in the next few days, especially looking forward for people to test it with entropix 🐸31 Oct 2024 19:35 +0000at://did:plc:ms5fdduxysmzdoplpbtbcbo4/app.bsky.feed.post/3l7tfpyrfdc2f