Overview

Though artificial intelligence (AI) has been discussed for decades, it has become a buzzword since OpenAI released ChatGPT in November 2022 (quite literally - The Economist deemed “ChatGPT” the word of the year for 2023). With well over a trillion dollars invested in AI companies and related infrastructure, and discussions around AI touching on ethics, economics, education, entertainment, energy, and more, it seems prudent to conclude our course by discussing how this new technology could impact cyberspace in the future.

An in-depth history, highly technical discussion, or robust ethical debate about AI is far beyond the scope of this course, and we will limit some of our focus to large language models (LLMs), which compromise many of today’s popular chatbots (ChatGPT, Gemini, Claude, Co-Pilot, etc.). Still, this lecture will provide a brief overview of AI terminology, history, and basic workings, before discussing vulnerabilities of models themselves, and how AI might be incorporated into cyber operations.

The most important takeaway should be that, for all its prowess, AI systems share some important similarities with all other software: they can be incredibly helpful and highly functional, but they can also be vulnerable, and the same underlying dynamics that make them powerful can be re-purposed by a malicious actor. Think back to our risk management lecture: as a general observation, for a constant cost, increasing the functionality of a system - which AI arguably does - also increases the risk of that system at the same time.

What is Artificial Intelligence?

Historical Origins

Artificial Intelligence (AI) as we know it today began in the 1950s, with some of the earliest work coming from pioneers like John McCarthy and Alan Turing. Turing considered whether computers truly think, and suggested the imitation game (now known as the Turing Test), where a human tries to distinguish a computer-generated response from a human-generated one. (Decades later, researchers furthered the concept with the proposed Lovelace Test, which examines whether a computer can fool a human on creative tasks.) Granted, passing the Turing Test may not indicate true understanding of the material presented, but it is considered an initial step towards intelligence.

The term "Artificial Intelligence" was first coined at the Dartmouth Conference in 1956, which is often considered the official birth of the field. Dr. McCarthy, a Turing award winner and often considered the "father of AI", suggested AI was the "science and engineering of making intelligent machines, especially intelligent computer programs".

AI development went through various ebbs and flows over ensuing decades, including multiple "AI winters" in which interest (and research funding) significantly waned. Still, progress did not stall - various AI systems were developed, but often with narrow focus for specific commercial applications. Some systems gained attention for their ability to play games; IBM's Deep Blue, for instance, was able to defeat a reigning world champion chess player in 1997. This lecture does not cover the full history of AI development, but for more understanding, consider checking out resources here, here, and here.

Recent Progress

It may feel like AI in any sort of practical application feels 'new', as some major developments within the field are less than 15 years old. In 2012, for instance, the field of deep learning experienced a breakthrough with AlexNet, a convolutional neural network architecture that won the ImageNet Challenge, an image recognition and classification competition. In 2014, the development of generative adversarial networks (GANs) helped pave the way for the generative AI (GenAI) used today. In 2017, researchers at Google introduced the concept of the transformer (the T in GPT) and the attention mechanism, whereby an AI system learns to concentrate on the most relevant portions of a text. While those developments may not have reached the general public, the rollout of ChatGPT certainly did: between November 2022 and April 2024, ChatGPT reached 100 million users, far faster than any previous product in history.

Artificial Intelligence Defined and Contextualized

Today, as part of the National Artificial Intelligence Initiative, the US legal code actually defines AI as such:

The term "artificial intelligence" means a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments. Artificial intelligence systems use machine and human-based inputs to-
(A) perceive real and virtual environments;
(B) abstract such perceptions into models through analysis in an automated manner; and
(C) use model inference to formulate options for information or action.

Other sources (businesses, academics, government bodies, etc.) use similar definitions, and some common threads are that AI systems can perform tasks like perception, reasoning, learning, and decision-making. This differs from more traditional computing tools, which follow fixed instructions; AI systems, on the other hand, can learn from experience to improve performance over time. Properly understood, AI is a rather large field, but many folks are interested in a specific subset known as deep learning, which is itself a subset of machine learning. Machine learning (ML) systems that improve their performance on a given task by learning from data, rather than being explicitly programmed with rules. Deep learning takes this a step further by using neural networks, which are algorithms modelled on the human brain. Today much of the hype surrounding AI involves generative AI (GenAI), which is able to create novel outputs, such as text, code, and imagery. While impressive, and likely able to pass the Turing Test, many still argue these systems don’t understand what they create, even if they produce output that appears very polished - in other words, they still have not achieved intelligence.

This leads into a brief aside about narrow versus general intelligence - otherwise known as Artificial Narrow Intelligence (ANI) and Artificial General Intelligence (AGI). ANI systems are tailored for specific applications; AGI, on the other hand, can generalize from context it has been trained on to perform tasks in new environments. AGI discussions don’t necessarily end there; some argue AGI implies a certain level of sentience, and perhaps even greater intelligence than humans. It is an open debate as to what AGI truly is and whether we will actually achieve it; some believe it will arrive by the end of the decade, while others argue we have already achieved it, at least by historical standards (other predictions exist as well).

AI and LLMS - How does it work?

We will avoid going further down the rabbit hole of AI definitions or delving into its impact on society, as fascinating as it may be, and pivot to a short overview of the technical side of AI systems. We will focus on Large Language Models (LLMs), which are trained on vast amounts of text data, such as internet webpages (as it turns out, AI firms may soon exhaust most useful text data from the internet). Chatbot models are trained to predict the next word in a conversation, generating conversational text as a result. The two videos below offer useful overviews of LLM functionality.

Among other observations, note that modern LLMs purposefully include some amount of randomness or variability. This makes the system far more interesting to work with, and far more likely to pass a Turing Test, but it also creates practical limitations. Some AI systems, for instance, can perfectly explain the process behind a mathematical calculation, but make errors in the calculation itself, as they treat calculations as a text generation issue instead of a mathematical one. Addressing this problem is an ongoing area of research - including for Trident Scholars at USNA!

AI Vulnerabilities

Like any software product, AI may be susceptible to vulnerabilities, and these can occur throughout its development (training and model creation) and deployment (e.g., usage among the target populace). For instance, model poisoning occurs when an attacker can manipulate the model itself, such as changing the weights assigned to certain parameters in a model. In a similar vein, data poisoning attacks occur when attackers inject malicious data into the training set, corrupting the AI’s learning process and potentially generating a model with biased or harmful results. An attacker might mislabel, manipulate, or inject data, such as marking spam or phishing emails as legitimate to compromise an email security system. Potentially, an attacker could even embed a 'backdoor' by training the model to perform a certain way when a unique situation is encountered. For instance, researchers have demonstrated how poisoned datasets can compromise self-driving cars: by adding small yellow stickers to stop signs, they caused AI systems to misclassify them as other signs, such as speed-limit signs. As a result, when the deployed model encountered sticker-altered stop signs in the real world, it failed to recognize them as stop signs. Injecting significant quantities of data may also bias the data towards an attacker's desired outcome.

In these cases, an attacker needs access to the data or development environment being used to create a model. There may be multiple ways for an attacker to achieve this,including more ‘traditional’ cyber operations in order to infiltrate an AI company’s development environment. (Think: an adversary compromises a developer’s account credentials, accesses a company’s infrastructure, and manipulates the training datasets.) So even if the product is a cutting edge AI tool, the basic cybersecurity elements discussed throughout the course still matter a great deal!

Once a model is deployed, attackers might switch to an abuse attack, where attackers insert incorrect or malicious information into the sources AI pull from, such as a website a chatbot queries to answer a user response. Even if the AI was trained on legitimate data, if it learns to query a certain website when responding to a user, an attacker might be able to impact an interaction by corrupting that queried website. Some deployed AI systems have access to outside content (emails or webpages), private data (passwords), and communication channels with the outside world - the "lethal trifecta", as researchers have coined it - allowing AI agents to exfiltrate sensitive data, among other attacks.

Attackers may also try to conduct evasion attacks on deployed systems, where AI models, especially in computer vision or natural language processing (NLP), are tricked by subtle, specially crafted inputs. These adversarial examples can cause the AI to make incorrect decisions, such as misidentifying an object or misleading an autonomous vehicle, posing security risks in critical applications. NIST gives the example illustrated in the graphic below, where deceptive markings on a street may cause an AI system to behave unexpectedly, with potentially deadly consequences.

Finally, in model inversion attacks (also known as privacy leaks), an attacker tries to gather sensitive information by interacting with the AI system, seeking to learn something about the data used to train a model, or how the model generates its output.

AI in Cyber Operations

We will conclude with a discussion of how AI could be incorporated into cyber operations (spoiler alert: it already is!), both from an offensive and defensive perspective. Just as an understanding of ports at the transport layer is necessary to enable communications, but also a vector for an attacker to gather information about a target system, AI can be a tool both for and against security in this field.

On the offensive side, AI has again many ways to make attacks more sophisticated. Consider a recently published report from Anthropic (owner of the Claude chatbot), who discovered a Chinese state-sponsored hacking group was using its chatbot to guide, create, and execute cyberattacks against ~30 targets. Some uses of AI for offensive operations include the following:

1. Spearphishing attacks - AI can be used to craft more convincing emails and even use social manipulation techniques. This is especially helpful when dealing with other languages.
2. Crafting malware - Many generative AI models have very capable coding ability making it easier for lower level programmers to make complex malware
3. Deepfake development - Generative AI can make very convincing deep fakes from anything from a fake video of your boss telling you to do something to a fake memo.
4. Automated vulnerability analysis - AI can be used to scope out a network to find out where it might be vulnerable and direct the hacker.
5. Botnet control - AI can be used to create and control botnets that can carry out massive tasks. Anything from exploiting simple boxes to organizing a Distributed Denial of Service attack (DDOS).

Conversely, AI could be used for defensive operations - in many cases, by performing the same activities as offensive operations, but using the findings to secure a system:

1. Patching software bugs - Just as AI can help attackers find vulnerabilities, it can also help defenders find and fix them. For instance, Google's Project Zero team used their Big Sleep AI agent to find a vulnerability in database software, allowing it to be patched before attackers could exploit it.
2. Threat Intelligence - Determining which threats to prioritize, predicting future attacks, and generating threat reports
3. Detecting and Investigating Malicious Activity - Anti-malware tools (anti-virus, EDR, IDS/IPS, etc.) can leverage AI to identify and investigate malicious files, traffic, and user activity.
4. Responding to Compromise – AI can streamline and automate incident response steps once a breach is detected. This might include isolating affected hosts, blocking malicious ccounts or IP addresses, rolling back malicious changes, or guiding analysts through recommended remediation steps. AI-driven playbooks can also adapt in real time as new evidence appears, speeding containment and reducing attacker dwell time.
5. Testing and Validation of Controls and Operations – AI systems can continuously test security controls, configurations, and operational workflows by simulating realistic attack behaviors. This includes generating synthetic adversary actions, fuzzing inputs, validating detection rules, and identifying gaps in monitoring or alerting pipelines. Regular automated validation helps ensure defenses remain effective as environments evolve.
6. Hunting for Active Threats – AI can support threat hunters by analyzing large volumes of logs, telemetry, and user behavior to surface anomalies and patterns that may indicate hidden attackers. Instead of relying solely on predefined rules, AI models can highlight unusual lateral movement, privilege escalation paths, rare process behaviors, or long-tail events that merit analyst review, accelerating proactive threat discovery.
7. Coordinating Cyber Defense – AI can help unify and orchestrate defensive actions across tools, teams, and environments. By aggregating data from disparate systems, correlating events, and recommending coordinated responses, AI can reduce silos and improve organization-wide situational awareness. In large or distributed environments, AI agents can also help synchronize patching, containment, policy updates, and communication during fast-moving security incidents.

---

Activity: Hack a Large Language Model!

Try your hand at hackmerlin.io! Can you get Merlin to hand over the secret password?

---