The Deep Dive Issue #12
The Bard is getting a fresh cut and making news as a potential rival to GPT-4, not a cheaper alternative mind you. While Gemini's performance can be erratic (at least they nailed the name), the promise remains in the potential from including AI into other Google services. As model performance starts to converge, the main differentiation factor may become the integration to third party platforms and devices, but only time will tell.
After our long awaited launch last week (we got tons of amazing feedback, thank you for engaging!), we resume this week with our compression blog posts series, this time diving into model pruning. Enjoy the read!
Need anything else?
Instruction tuning is used to improve a pre-trained model's ability to handle instruction inputs. Despite proliferating instruction datasets, open source instruction processing tools remain scarce and model-specific. EasyInstruct offers a general instruction processing framework for LLMs that can handle generation, selection, prompting, and the interactions between these components of the instruction tuning pipeline.
Why would you care? - If you're the type to experiment with lots of models, chances are you've stopped trying to understand why instruction templates look that way. If it doesn’t make sense, might as well automate it.
How does it work? Instruction tuning involves refining a pre-trained LLM's response by using specific prompting formats such as (INSTRUCTION, INPUT, OUTPUT), where INSTRUCTION is a directive that outlines the task, INPUT offers additional context, and OUTPUT specifies the user's expected outcome. This general format can vary depending on the tuning method with each method being more appropriate to a given task.
EasyInstruct automates the selection and use of these templates via four building blocks modules:
Engines: Integrate LLM APIs from providers such as OpenAI and Anthropic, as well as some open-source models, and standardize the instruction execution process across models.
Generators: Automatically generate the proper instruction based on seed data (Chat, Corpus, or Knowledge Graph) or desired instruction method such as Self-Instruct, Evol-Instruct, Backtranslation or KG2Instruct.
Selectors: Curate the instruction datasets based on a set of metrics that evaluate the quality of instructions such as an LLM-based perplexity score, an n-gram based score, or a statistical textual diversity-based score.
Prompts: Standardize the instruction prompting step where requests are constructed as instruction prompts and sent to specific LLMs.
Further, EasyInstruct comes equipped with various prompting techniques that can be called in a standardized manner. For e.g you can use Chain-of-Thoughts prompting on Llama2 with your base prompt as follows:
from easyinstruct import ZeroshotCoTPrompt
from easyinstruct import Llama2Engine, ChatGLM2Engine
question = """
I used 500$ in API credits in April,
and then used half that amount in May.
I'm I spending too much in deployment?
""" # unify.ai/hub much?
zeroshot_prompts = ZeroshotCoTPrompt() # or other techniques
zeroshot_prompts.build_prompt(question)
engine = Llama2Engine( # or other engines
pretrained_model_name_or_path="../llama2-7b-chat"
)
print(
zeroshot_prompts.get_engine_result(engine = engine)
)Check out the repository to get started.
The Lab
Children morals
Improving a model's alignment through reinforcement learning (RL) requires high quality, hard to obtain feedback from human annotators, and involves coordinating multiple models to train, process, and evaluate answers.
Alternatively, Aligner only requires an extra module, making it pluggable and usable with API-based models. It uses weak-to-strong generalization by integrating a small Aligner model with strong models to supervise expert LLMs.
Aligner takes the user’s query and initial answer generated by the upstream LLM to generate an answer better aligned with human values. The answers are then used to fine-tune the model for better alignment. Focusing on alignment correction reduces capacity needs and allows using a small Aligner model.
Results reveal better LLM performance in terms of helpfulness and harmlessness, across many tasks.
If you run it, put a label on it
LLMs perform poorly on domain-specific tasks, and solutions have their own issues. For e.g fine tuning is prone to catastrophic forgetting, in-context learning is less effective, and training specialized LLMs is costly.
Tag-LLM improves an LLM's performance on specialized tasks by better using its learned representations. It adds a set of trainable ‘tags' to the input to steer the LLM's distribution when specific vocabulary appears.
Domain tags are inserted in front of domain-specific language (e.g protein sequences) and trained using unlabeled data.
Function tags guide the model's use of domain inputs to solve specific tasks (e.g translation) and are trained using labelled task-specific data.
Tag-LLM first introduces domain tags to prefix specialized data, before gradually appending function tags to the end which are learned by directly optimizing the downstream task loss. For multi-domain prompts, domain tags are frozen and the tasks are trained to optimize the loss function on all domain datasets.
By disentangling domains from task functions, Tag-LLM enables effective zero-shot generalization to unseen problems.
More cutting-edge research
A reasonable approach - Self-Discover is a new prompting framework that applies the best prompting method for a task. Self-Discover uses a language model to find a subset of reasoning modules (Chain of Thoughts, Zero-shot prompting, etc.), adapt the general instruction to the task (e.g "calculate each operation" instead of "break down the problem" for an arithmetic problem) and uses this tailored set of instructions to achieve better average performance than any singular prompting method.
Classroom of teachers - Current knowledge transfer methods lack knowledge diversity as they use a single teacher model, and only use ground truth data as prediction targets. Instead, TinyLLM employs multiple teacher LLMs to generate the rationale behind their answers using in-context examples, and train a student model not only to predict labels but also generate similar rationale. TinyLLM improves the student models' reasoning ability and can achieve up to +12.57% of performance improvement with 1.1% of model size.
See what I see? - LEVI combines views of different models to attenuate the impact of spurious features on a fine-tuned model's ability to generalize to new input distributions. Specifically, it connects the intermediate outputs of a large model with the final output of a smaller model using adapting layers whose outputs are used as predicted labels. Experiments on large language and vision models show improved fine-tuning out-of-distribution generalization performance without compromising on in-distribution performance.
The Pulse
Sam-bots - A report in The Information states that OpenAI is creating smart software agents to improve devices' ability to perform tasks independently. These agents will be able to gather company data and arrange travel plans, among other things. The goal is for these agents to become more adaptable and effective over time, eventually handling a wider range of tasks.
Seal of inauthenticity - OpenAI has started adding C2PA metadata to images created by DALL·E 3, which includes details about the origin of the image. Similarly, Meta is also introducing "Imagined with AI" labels on AI-generated images across its platforms. These moves aim to help users determine the authenticity of the images and provide transparency regarding their creation.
Move fast but move safe - The UK government has announced over £100 million in new funding to promote agile AI regulation. They plan to introduce future binding requirements for developers of advanced AI systems to ensure safety and accountability. These measures aim to strengthen the UK's reputation as a leader in AI safety and development while addressing public concerns regarding AI risks.
And that’s all for this edition, we hope you enjoyed reading through!
The Unify Dev Team.