The power of machine learning is being leveraged to solve increasingly complex problems across a range of different areas. Models are required to recognise and understand more abstract concepts and objects, and in many cases make non-linear decisions. Although powerful in their own right, the more traditional types of machine learning models lack the ability to accurately map and process some of the most complex problems.
Increasingly, artificial neural networks are recognised as providing the architecture for the next step in machine learning. These networks are designed to mirror the functionality of the human brain and nervous system. A web of connected nodes act as artificial neurons, and deep learning techniques are used to create models which can make non-linear decisions. However, the majority of models are designed to process euclidean input data, with a grid-like structure over one or two dimensional domains. This includes common representations of file types like images or text. But there is non-euclidean data that might exist over three dimensions, such as maps of networks or molecules. Valuable information will be lost if this data is converted to euclidean data structure.
Graphs are a common type of non-euclidean data structure, an abstract way of modelling data which consists of nodes and interconnecting edges. Graph Neural Networks are artificial neural networks developed to process graph structured input data. Graph Neural Networks can be seen as a subdomain of a wider field of geometric deep learning. This guide will explore the concept of Graph Neural Networks, including what they are, how they’re built, the challenges the field faces, and the different applications of Graph Neural Networks.
What is a Graph Neural Network?
Graph Neural Networks are a type of artificial neural network which are designed to process graph structured data. They form part of a broader field of artificial neural networks, which are networks designed to reflect the functions of human or animal brains. There are many different types of artificial neural network, but they are generally unified by the aim of artificially replicating the behaviour of neurons and synapses. Networks are usually grouped into layers of nodes or artificial neurons, connected with other nodes. These layers exist between an input and an output layer, through which the data travels.
Artificial neural networks are seen as a powerful way of creating machine learning models, which can understand complex concepts and patterns beyond traditional model architecture. Graph Neural Networks are unique because they process graph structured data, in contrast to many machine learning models that process euclidean data with a grid-like structure. Most of the most common types of data that are processed by machine learning models will be classed as euclidean data. This is data that exists in one or two-dimensional domains, for example text, audio files, and images.
In contrast, graph structured data can exist in a three dimensional domain, and is a key example of non-euclidean data. Graph data structures can be used to more accurately represent complex concepts. This could include molecules or networks where a third dimensional domain is important. Valuable information may be lost when this complex data is transformed into a euclidean structure. Most things can be represented by a graph structure, so Graph Neural Networks have a huge potential array of applications.
There are a range of different Graph Neural Networks, which can be understood as an artificial neural network that is capable of processing graph structured data. The choice of Graph Neural Network will hinge on the task at hand, as well as the type and complexity of graph that requires processing.
Four popular examples of Graph Neural Networks includes:
- Recurrent graph neural networks (RecGNN)
- Convolutional graph neural networks (ConvGNN)
- Graph autoencoders (GAE)
- Spatial-temporal graph neural networks (STGNN)
What is graph structured data?
It’s important to grasp the basics of graph structured data to fully understand a Graph Neural Network. Graphs are abstract, non-euclidean structures consisting of a set of points connected by links or edges. Graphs have a defined amount of points or nodes, with the edges or links connecting pairings of nodes. Graphs therefore provide information on the relationships between different nodes as well as the overall structure of features.
The different edges or lines can be labelled or hold a value. Graph structured data can be ingested into models built on Graph Neural Networks, which in turn can perform tasks such as clustering, regression or classification. There are a range of different types of graphs, each with different properties.
The main types of graphs are:
- Directed graphs. The lines or edges have a defined direction
- Undirected graphs. The lines or edges have no specific direction.
- Weighted graphs. Value, labels or weighting is added to each line or edge.
- Cyclic graphs. A graph which includes a circle of connected nodes.
A huge range of things can be modelled as graph structured data. For example, train stations can be the nodes and the lines that connect them to the edges. A more abstract example would be within reinforcement learning, which is a process in which a model learns through a trial and error feedback loop. The model will perform actions to transition between states in an environment. This can be modelled as a graph in which states are nodes and actions are edges.
Examples of chart structured data includes:
- Molecular structures, in which bonds are links and atoms nodes.
- Social networks, in which users are nodes and social connections are edges.
- Artificial neural networks, in which artificial neurons are nodes and the connections are edges.
- Country maps, in which cities and towns are nodes and roads are edges.
Applications of Graph Neural Networks in machine learning
Machine learning models often face issues when processing graph data, because of the sheer size of many graph structures and the web-like structure of the graph. A graph modelled on big data such as social networks can have billions of interlinking nodes, making it incredibly difficult to process. In addition, the different types of graphs will have different structures in space, making the approach to processing nodes an abstract problem. They’re a web of interconnected nodes, so a clear start and end won’t exist.
For traditional machine learning approaches, graphs must be transformed into vectors before being processed. This preprocessing can cause relevant information on the relationship between nodes to be lost. Graph Neural Networks are therefore used in machine learning to process graphs directly. Graph Neural Networks can then make predictions about the node or edges in graphs.
Models built on Graph Neural Networks will have three main focuses:
- Tasks focusing on nodes. Examples could include clustering or classifying nodes, or performing regression to predict a continuous value. Classification is the labelling of nodes into different categories, and clustering would be the segmenting of nodes into similar groupings.
- Tasks focusing on edges or lines. These represent the relationships between nodes, so generally this type of task will include classification on the type of edge or predicting the connection between nodes.
- Tasks focusing on the graph-level. This would include predictions and classification of the graph as a whole.
Because most things can be modelled using graph structured data, Graph Neural Networks have a huge range of applications. Graph-level tasks can include clustering and matching different graphs. This could mean clustering different diseases or viruses, or matching molecular structures. Node-level tasks such as node classification have a huge array of applications too. Text, voice and image files can all be classified using Graph Neural Network model architecture. Recommendation systems and text context extraction are both examples of edge-level tasks, which play a key role in approaches to translating sequences and sentences.
Graph Neural Networks have a range of applications, for example:
- Generating user recommendations based on relationships between different products previously purchased
- Recommending new connections in social media networks.
- Predicting shared interests of users within ecommerce or social media networks.
- Predicting changes in ecosystems or interconnected diseases, or the mutation of viruses.
- Classification of objects, or the labelling of unlabelled data through different nodes assigned a different object label.
- Processing modelled scientific systems and structures, with applications in physics, chemistry and biology.
- Processing of knowledge graphs which map relationships between events, locations, concepts or entities.
- Processing user interactions with other users, items and systems.
- Labelling of sequences and the extraction and translation of text.
How are Graph Neural Networks built?
There are many different examples of Graph Neural Network models, but the most common approaches will use different combinations of similar modular parts. The final model pipeline will be reflective of the network’s application, the type and scale of graph structured data it will process, and the type of task the model will perform.
The general process for building a Graph Neural Network will be to define the complexity and scale of the graph data and the type of task the model will complete. The architecture of the Graph Neural Network can then be designed and built, usually from a combination of modules which deal with unique parts of graph structured data. For example a module which is used to sample large, complex graphs to make processing possible.
The first stage is to clearly define the structure of the graph data that needs to be processed. This could be a specific structure as with a molecular model, or a graph that needs to be generated from non-structured data. The type of graph should then be defined, to understand its complexity and scale. For example, establishing whether the graph edges are directional or not will highlight the graph’s level of complexity. The size and scale of the graph needs to be determined. Some of the more complex graphs processed by Graph Neural Networks such as social media platform connections can have billions of nodes. These huge graphs aren’t easy to access, store, or process, so Graph Neural Networks will need to include data sampling processes.
Next the task the Graph Neural Network model will be performing should be confirmed, so that the correct loss function can be created. Examples of tasks could be clustering of nodes from chart structured data, or the prediction of edges and connections between nodes. Once this baseline information is determined, the model’s architecture can be designed and built.
A common pipeline design includes combinations of different modules which deal with the processing of unique properties of graph data. These modules exist as layers in the Graph Neural Network dataflow. For examples where the graph is huge as with big data like in social media, a sampling module may be required to make processing possible. Likewise, a pooling module might be included to reduce the complexity of graphs.
The challenges of Graph Neural Networks
Graph Neural Networks can be leveraged to create powerful models which can achieve complex tasks beyond traditional machine learning techniques. But Graph Neural Networks face a range of problems and challenges shared across the machine learning field, as well as unique challenges in the graph domain.
A major challenge is in the pretraining and training phase of creating a Graph Neural Network. Whereas a traditional machine learning technique may need hundreds of thousands of data points to be trained, Graph Neural Networks may need millions or billions of training data points. Graph Neural Network models generally need labelled training data, which is labelled by humans so is an extremely resource-intensive process. Techniques in transfer learning are usually leveraged to lower the intensity of training required.
Graph Neural Networks are black box models, which means it’s difficult to adequately explain the decision-making process of the model. The challenge is one of model explainability, which is important once a model is deployed to a real-world environment. Decisions made by models may not be fully trusted if they cannot be explained to a human. As Graph Neural Network models have strong applications in medicine and science research, explainability is a focus and challenge.
Finally, the complexity of the graph structured data is an intrinsic challenge for Graph Neural Networks. As models designed for the graph domain, the complexity of graph structures are a constant consideration and challenge. There are many different types of graph structures which range in scale and complexity. More powerful Graph Neural Networks are required as more complex graph structures emerge, such as big data from social media platforms. There’s more data than ever before, and the complexity of graph structured data models are increasing exponentially. Dynamic graph structures add another layer of challenge that is a focus of research too.
Machine learning deployment for every organisation
Seldon moves machine learning from POC to production to scale, reducing time-to-value so models can get to work up to 85% quicker. In this rapidly changing environment, Seldon can give you the edge you need to supercharge your performance.
With Seldon Deploy, your business can efficiently manage and monitor machine learning, minimise risk, and understand how machine learning models impact decisions and business processes. Meaning you know your team has done its due diligence in creating a more equitable system while boosting performance.
Deploy machine learning in your organisations effectively and efficiently. Talk to our team about machine learning solutions today.