A new paper by Meta shares how to extend context windows via positional interpolation.

I am still reading to confirm this, but as far as I understand this topic - this is the same method based on kaiokendev’s approach with SuperHOT context lengths.

If you want to learn more about how 8k Context w/ SuperHOT was recently achieved (beyond the paper Meta shared), I highly recommend visiting kaiokendev’s pages and posts below.

• https://kaiokendev.github.io/

• https://kaiokendev.github.io/til

• https://kaiokendev.github.io/context

I was curious to hear more about SuperHOT myself, so I emailed kaiokendev and asked for learning material suggestions.

Here is what they shared with me. Thank you for this list, kaiokendev!

Recommendations from the Developer of SuperHOT (kaiokendev):

Here are some resources to help with learning LLMs:

Andrej Karpathy’s GPT from scratch:

• https://www.youtube.com/watch?v=kCc8FmEb1nY

Huggingface’s NLP Course:

• https://huggingface.co/learn/nlp-course/chapter1/1

And for training specifically:

Alpaca LoRA:

• https://github.com/tloen/alpaca-lora#-alpaca-lora

Vicuna:

• https://github.com/lm-sys/FastChat#fine-tuning

Community training guide:

• https://rentry.org/llm-training

Of course for papers, I recommend reading anything on arXiv’s CS - Computation & Language that looks interesting to you：https://arxiv.org/list/cs.CL/recent.

If you found this post interesting, please consider subscribing to the /c/FOSAI community at !fosai@lemmy.world where I do my best to keep you in the know with the most important updates in free open-source artificial intelligence.