Model is trained on his own orca style dataset as well as some airoboros apparently to increase creativity
Quants:
https://huggingface.co/TheBloke/dolphin-2.0-mistral-7B-GPTQ
Model is trained on his own orca style dataset as well as some airoboros apparently to increase creativity
Quants:
https://huggingface.co/TheBloke/dolphin-2.0-mistral-7B-GPTQ
It depends on the learning rate, typically it’s ideal and higher quality to learn really slowly over a lot of epochs but it’s cheaper and obviously faster to learn fast over fewer epochs
Also the dataset size is important to consider
I was concerned that a large dataset with low sentence similarity may take longer to train. I’m not sure if my idea that novels take less time to train than a Q&A dataset with detailed answers is true: generic roleplay vs encyclopedic knowledge.
Reading these datasets, I think these GPT3/4 conversations go into too much detail, and current (1-40B) language models cannot be trained in such detail. These conversations would be only useful for humans. But I might be wrong about training because I don’t have experience with 100B+ models, and how they scale down.