Get the latest tech news
StarCoder2-Instruct: Transparent Self-Alignment for Code Generation
StarCoder2-Instruct: Fully Transparent and Permissive Self-Alignment for Code Generation - bigcode-project/starcoder2-self-align
We introduce StarCoder2-15B-Instruct-v0.1, the very first entirely self-aligned code Large Language Model (LLM) trained with a fully permissive and transparent pipeline. Our open-source pipeline uses StarCoder2-15B to generate thousands of instruction-response pairs, which are then used to fine-tune StarCoder-15B itself without any human annotations or distilled data from huge and proprietary LLMs. Optimizer: Adafactor Learning rate: 1e-5 Epoch: 4 Batch size: 64 Warmup ratio: 0.05 Scheduler: Linear Sequence length: 1280 Dropout: Not applied
Or read this on Hacker News