CISO Perspective: 10 Critical Cybersecurity Considerations for the GenAI Era

The LLM AI Cybersecurity & Governance Checklist aims to enable leaders to quickly identify key risks associated with GenAI and LLMs and equip them with associated mitigations. OWASP stresses the checklist isn’t exhaustive and will shift as the GenAI and LLM usage progresses and the tools themselves develop and mature.

The checklist starts off with term clarification. Generative AI is defined as a type of machine learning (ML) that focuses on creating new data, while LLMs are a type of AI model used to process and generate human-like text.

There are various threat categories, captured in the image below:

The checklist also lays out a six-step practical approach for organizations to develop their LLM strategy:

There are also various LLM deployment types, each with unique considerations. The types range from public application programming (AP) access and licensed models, all the way to custom models:

With those key considerations out of the way, let’s walk through the checklist areas and offer key takeaways from each.

This area involves competitors and attackers and is focused not only on the attack landscape, but also on the business landscape. This includes understanding how competitors are using AI to drive business outcomes, as well as updating internal processes and policies such as incident response plans (IRPs) to account for GenAI attacks and incidents.

The adage “you can’t protect what you don’t know you have” applies to GenAI and LLMs. This checklist area involves having an AI asset inventory for both internally developed offerings and external tools and platforms. It’s crucial to understand not only the tools and services being used by the organization, but also the “ownership” in terms of who will be accountable for their use.

An organization can properly integrate AI security and privacy training into its GenAI and LLM adoption journey. Doing so involves helping staff understand existing GenAI/LLM initiatives, as well as the broader technology, how it functions, and key security considerations, such as data leakage.

Further, it’s essential to establish a culture of trust and transparency so staff feel comfortable sharing what GenAI and LLM tools and services are being used and how. Building this trust and transparency within the organization is crucial to avoiding “shadow AI” usage. Otherwise, people will continue to use these platforms without informing IT and security teams for fear of consequences or punishment.

This one may be surprising, but much like with cloud before it, most organizations don’t establish coherent strategic business cases for using new innovative technologies, including GenAI and LLM. It is easy for businesses to get caught in the hype and feel they need to join the race or get left behind. But without a sound business case, the organization risks poor outcomes, increased risks, and opaque goals.

Without governance, accountability and clear objectives are nearly impossible. This area of the checklist involves establishing an AI RACI (responsible, accountable, consulted, and informed) chart for the organization’s AI efforts, documenting and assigning who will be responsible for risks and governance and establishing organizational-wide AI policies and processes.

This area involves an extensive list of activities, such as product warranties involving AI, AI EULAs (end user license agreements), ownership rights for code developed with AI tools, IP risks, and contract indemnification provisions. Businesses should be sure to engage their legal team or experts to determine the various legal-focused activities the organization should be undertaking as part of their adoption and use of GenAI and LLMs.

Using LLM tools requires specific risk considerations and controls. The checklist calls out items such as access control, training pipeline security, mapping data workflows, and understanding existing or potential vulnerabilities in the LLM models and supply chains. Additionally, there is a need to request third-party audits, penetration testing, and even code reviews for suppliers, both initially and on an ongoing basis.

To ethically deploy LLMs, the checklist calls for the use of model and risk cards, which can be used to let users understand and trust the AI systems as well to openly address potentially negative consequences such as biases and privacy. These cards can include items such as model details, architecture, training data methodologies and performance metrics. There is also emphasis on accounting for responsible AI considerations and concerns around fairness and transparency.

Retrieval-augmented generation (RAG) is a way to optimize LLM capabilities when it comes to retrieving relevant data from specific sources. It is a part of optimizing pre-trained models or re-training existing models on new data to improve performance. The checklist recommended implementing RAG to maximize the value and effectiveness of LLMs for organizational purposes.

Lastly, the checklist calls out the use of AI red teaming, which is emulating adversarial attacks of AI systems to identify vulnerabilities and validate existing controls and defenses. It does emphasize that red teaming alone isn’t a comprehensive answer or approach to securing GenAI and LLMs but should be part of a comprehensive approach to secure GenAI and LLM adoption.

That said, it is worth noting that organizations need to clearly understand the requirements and ability to red team services and systems of external GenAI and LLM vendors to avoid violating policies or finding themselves in legal trouble.

AI red teaming and penetration testing is also called for in other sources such as those by NIST discussed above, as well as in the EU AI guidance

While not exhausting all potential GenAI and LLMs threats and risk considerations, the OWASP LLM AI Cybersecurity & Governance Checklist represents a concise resource for organizations and security leaders. It can aid practitioners in identifying key threats and ensuring the organization has fundamental security controls in place to help secure and enable the business as it matures in its approach of adopting GenAI and LLM tools, services, and products.