Democratizing Large Language Model development with InstructLab support in watsonx.ai
There is no doubt that generative AI is changing the game for many industries around the world due to its ability to automate and enhance creative and analytical processes. According to McKinsey, generative AI has a potential to add $4 trillion to the global economy. With the advent of generative AI and, more specifically, Large Language Models (LLMs), driving tremendous opportunities and efficiencies, we’re finding that the path to success for organizations to effectively use and scale their generative AI projects in production is not that straight forward—with only 10% of companies today having put their gen AI solutions into production according to Gartner.
The next iteration of Language Models
Scaling AI requires investment in your company, talent, infrastructure and data. We are entering the era where quality datasets to train the LLMs are needed more than ever before. Many companies are looking to adapt and tune models with their proprietary data to teach the model the language of their business. Further, model architectures are changing. They are becoming increasingly modular, relying on external memory. As such, organizations are taking a multimodel approach, where they want the flexibility and modularity to work with various models, whether it be open-source or commercial, depending on the use case.
Governance is critical. You need to govern and monitor models for trust, transparency and end-to-end explainability, including tracking models for any hallucinations, bias, drift and more. Finally, we’re seeing the emergence of AI agents as the main experience layer for how consumers interact and chat with generative AI applications today with end-to-end task automation.
With all these changes, there is a common thread—how do we keep up? Because LLMs might periodically become outdated by lacking the latest knowledge and skills there are not many well-known ways in which AI developers can effectively contribute and enhance a single LLM in the open. We’ve observed that today’s LLMs have not been truly community-driven, and there is no clear path to contributing and enhancing an existing LLM in the open to get the full benefits of community-driven development.
This has resulted in developers creating multiple forks, or variants of an LLM, that cater to different specializations. With lack of standards, many of today’s “open-source LLMs” can suffer from monolithic development with siloed contributions, where no one knows what’s coming or how to best train and tune the model for their wanted task. At IBM, we believe in a better path forward to help enable real collaborative model development in the open—to democratize AI building for everyone.
Introducing InstructLab: Disrupting model customization
To overcome the challenge of multiple forks of a single open-source model, IBM Research® released a new methodology called called LAB (Large-Scale Alignment for ChatBots). LLMs behind modern-day chatbots are pretrained on large sets of unstructured text, which allows them to learn many new tasks quickly once they see labeled instructions during the alignment stage. However, creating quality instruction data is difficult and costly to come by. Hence, this LAB technique aims to overcome some challenges around LLM training by using taxonomy guided synthetic data generation. And that’s why we introduced InstructLab—an open-source project by IBM Research and Red Hat® that builds on the LAB technique for a community-driven approach to language model development through skills and knowledge training.
InstructLab offers a novel method for collaborative customization and tuning of LLMs. The toolkit systematically generates synthetic data for tasks that you want chatbots to accomplish, and for assimilating new knowledge and capabilities into the foundation model—without overwriting what the model has already learned. With InstructLab training techniques, LLMs can be improved in less time and at a lower cost than is typically spent training LLMs. To power this community engagement, developers can get access to the seed compute and infrastructure to train the projects’ LLMs, with continuously updated and released models that merge all accepted community contributions on a weekly cadence orchestrated from Red Hat.
Our new LAB-aligned models achieve state-of-the-art chat performance
Based on internal benchmark testing, we’ve seen great performance with the InstructLab technique on the MT-Bench score for IBM’s Granite-chat-13B-v2 model in watsonx.ai™. IBM Research also found that when applying the InstructLab method to the open-source LLM Merlinite, which is built on Mistral 7B, it achieved strong scores on MT-Bench and MMLU (5-shot).
Developers can access a series of InstructLab tuned language models and open code models through watsonx.ai. This includes the release of four InstructLab trained language models (granite-7b-lab, merlinite-7b, granite-20b-multilingual and granite-13b-chat-v2) in addition to four state-of-the-art open code models (granite-3b, granite-8b, granite-20b and granite-34b) that perform well across a range of coding tasks, including code generation and fixing.
Through the InstructLab project, IBM and Red Hat have released select open-source licensed Granite language and code models under the Apache 2.0 license. Through subscribing to the commercial license of RHEL AI from Red Hat or by accessing the InstructLab models and toolkit in watsonx.ai, clients can get access to these open-source-licensed Granite language and code models that are also supported and indemnified by Red Hat. For instance, IBM’s granite-7b English language model is now fully integrated into the InstructLab community and accessible directly in watsonx.ai, where developers can contribute new skills and knowledge to collectively enhance these models, just as they would to any other open-source project.
watsonx.ai has also introduced a new interactive visualization called “Taxonomy Explorer” that allows users to explore the knowledge, skills data and taxonomy behind an InstructLab model tuning.
Providing a platform for collaborative Language Model enhancements to build the future of generative AI
We’re continuously evolving our model strategy in watsonx.ai to help enterprise developers and line-of-business (LoB) leaders accelerate AI application development. This requires a full-stack approach for the open-hybrid cloud to scale generative AI capabilities, tools and platforms, to succeed in the rapidly evolving digital landscape. In watsonx.ai, we’ve built an intuitive, collaborative development studio environment with prebuilt generative AI patterns, while integrating essential enterprise features for production level workloads. For instance, enterprise developers can optimize the development of production-ready AI applications with models, tools, SDKs, Notebooks, API integrations and runtimes to deploy AI applications at scale.
At IBM, we are committed to fostering an open innovation ecosystem around AI to help our clients maximize model flexibility and enhancements with new skills and knowledge. As part of our hybrid, multimodel strategy, we’ll continue to offer a mix of third-party models from strategic partners, such as Meta and Mistral AI, as well as select open-source models from Hugging Face, bring-your-own models (BYOM), in addition to proprietary, domain-specific IBM-developed models with IP indemnification, as well as IBM open-sourced InstructLab code and language models licensed from Red Hat. Our open, multimodel, multilingual strategy is continuing to take shape as highlighted in our most recent strategic collaboration with the Spanish government that aims to build the world’s leading suite of foundation models, including both large language models and small language models, proficient in the Spanish language and co-official languages.
Through our close partnership with Red Hat, in the future we intend to embed the supported InstructLab alignment CLI from RHEL AI’s foundation model runtime engine directly in watsonx.ai to support end-to-end developer AI workflows to quickly adapt models with new skills and knowledge using proprietary business data, all within a single studio interface. This might help facilitate faster deployment, tuning and customization of open and custom InstructLab trained models in watsonx.ai to then scale those models across machines, devices, end-applications and business processes. Further, organizations might benefit from native integrations to the rest of the watsonx platform, to build, scale and govern their AI solutions with data lineage, storage and lifecycle governance – across any cloud or on-premises environment.
Ready to learn more?
To learn more about the InstructLab project from IBM Research and Red Hat, visit the GitHub page and get started contributing to the community. Book a meeting if you’re interested to know more about watsonx.ai—IBM’s next generation enterprise studio for AI builders to train, validate, tune and deploy AI models—or begin working with the InstructLab trained language and code models and other foundation models in our library, by signing up for a free trial of watsonx.ai.
Begin working with InstructLab trained language and code models
Statements regarding IBM’s future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
