An introduction to Physics Informed Neural Networks (PINNs): Teach your neural network to “respect” Physics

https://preview.redd.it/ll4z0ewvqwdg1.png?width=1100&format=png&auto=webp&s=e6a375679fb5575866953109c00e86d8eb31523a As universal function approximators, neural networks can learn to fit any dataset produced by complex functions. With deep neural networks, overfitting is not a feature. It is a bug. Medium Link for better readability: https://vizuara.medium.com/an-introduction-to-physics-informed-neural-networks-pinns-teach-your-neural-network-to-respect-af484ac650fc Let us consider a hypothetical set of experiments. You throw a ball up (or at an angle), and note down the height of the ball at different points of time. When you plot the height v/s time, you will see something like this. https://preview.redd.it/b9byjx62pwdg1.png?width=1100&format=png&auto=webp&s=22aebc098ad30d2b18505fcaa3d80cf61777f2b5 It is easy to train a neural network on this dataset so that you can predict the height of the ball even at time points where you did not note down the height in your experiments. First, let us discuss how this training is done. # Training a regular neural network https://preview.redd.it/732wrp23pwdg1.png?width=1100&format=png&auto=webp&s=5c65e4fc46e3a8fd8fcac281361ece4328932f2b You can construct a neural network with few or multiple hidden layers. The input is time (t) and the output predicted by the neural network is height of the ball (h). The neural network will be initialized with random weights. This means the predictions of h(t) made by the neural network will be very bad initially as shown in the image below. https://preview.redd.it/xdgeu9s4pwdg1.png?width=1100&format=png&auto=webp&s=2e97b932fe7bef937f45716295435c7d50c0212f We need to penalize the neural network for making these bad predictions right? How do we do that? In the form of loss functions. Loss of a neural network is a measure of how bad its predictions are compared the real data. The close the predictions and data, the lower the loss. A singular goal of neural network training is to minimize the loss. So how can we define the loss here? Consider the 3 options below. https://preview.redd.it/slcx6y27pwdg1.png?width=1100&format=png&auto=webp&s=fcccb9ec6c9aac8b976b71ae5a7f7f6dfd481c24 In all the 3 options, you are finding the average of some kind of loss. * **Option 1 is not good** because positive and negative errors will cancel each other. * **Option 2 is okay** because we are taking the absolute value of errors, but the problem is modulus function is not differentiable at x=0. * **Option 3 is the best**. It is a square function which means individual errors are converted to positive numbers and the function is differentiable. This is the famous Mean Squared Error (MSE). You are taking the mean value of the square of all individual errors. Here error means the difference between actual value and predicted value. Mean Squared Error is minimum when the predictions are very close to the experimental data as shown in the figure below. https://preview.redd.it/vwm6mxq8pwdg1.png?width=1100&format=png&auto=webp&s=33983e165ecec1efca3a973e97b3d28aa2a89782 But there is a problem with this approach. What if your experimental data was not good? In the image below you can see that one of the data points is not following the trend shown by the rest of the dataset. https://preview.redd.it/mswknvl9pwdg1.png?width=1100&format=png&auto=webp&s=71546cc05f741175a11e486ae3fe6a77c44b82e7 There can be multiple reasons due to which such data points show up in the data. 1. You did not perform the experiments well. You made a manual mistake while noting the height. 2. The sensor or instrument using which you were making the height measurement was faulty. 3. A sudden gush of wind caused a sudden jump in the height of the ball. There could be many possibilities that results in outliers and noise in a dataset. Knowing that real life data may have noise and outliers, it will not be wise if we train a neural network to exactly mimic this dataset. It results in something called as overfitting. https://preview.redd.it/1e7r509apwdg1.png?width=1100&format=png&auto=webp&s=e3269c58b8ca9e873945ca9970aafac78bc53279 https://preview.redd.it/l0fgrzrapwdg1.png?width=1100&format=png&auto=webp&s=28acb46d2af8e6398876ee107b7900e860061904 In the figure above, mean squared error will be low in both cases. However in one case neural network is fitting on outlier also, which is not good. So what should we do? # Bring physics into the picture If you are throwing a ball and observing its physics, then you already have some knowledge about the trajectory of the ball, based on Newton’s laws of motion. Sure, you may be making simplifications by assuming that the effect of wind or air drag or buoyancy are negligible. But that does not take away from the fact that you already have decent knowledge about this system even in the absence of a trained neural network. https://preview.redd.it/8cudgx0epwdg1.png?width=1100&format=png&auto=webp&s=9efaf22e50525030c0ceaa9995b0afe96a26c79d The physics you assume may not be in perfect agreement with the experimental data as shown above, but it makes sense to think that the experiments will not deviate too much from physics. https://preview.redd.it/fpy7q3oepwdg1.png?width=1100&format=png&auto=webp&s=dc5ff5cacaf8b8d2895139589897c6dd3d670be9 So if one of your experimental data points deviate too much from what physics says, there is probably something wrong with that data point. So how can you let you neural network take care of this? # How can you teach physics to neural networks? If you want to teach physics to neural network, then you have to somehow incentivize neural network to make predictions closer to what is suggested by physics. If the neural network makes a prediction where the height of the ball is far away from the purple dotted line, then loss should increase. If the predictions are closer to the dotted line, then the loss should be minimum. What does this mean? Modify the loss function. How can you modify the loss function such that the loss is high when predictions deviate from physics? And how does this enable the neural network make more physically sensible predictions? **Enter PINN Physics Informed Neural Network.** Physics Informed Neural Network (PINN) The goal of PINNs is to solve (*or learn solutions to*) differential equations by embedding the known physics (or governing differential equations) directly into the neural network’s training objective (loss function). The idea of PINNs were introduced in this seminal paper by Maziar Raissi et. al.: https://maziarraissi.github.io/PINNs/ The basic idea in PINN is to have a neural network is trained to minimize a loss function that includes: 1. A **data mismatch** term (*if observational data are available*). 2. A **physics loss** term enforcing the differential equation itself (and initial/boundary conditions). # Let us implement PINN on our example Let us look at what we know about our example. When a ball is thrown up, it trajectory h(t) varies according to the following ordinary differential equation (ODE). https://preview.redd.it/vacsz6dlpwdg1.png?width=1100&format=png&auto=webp&s=14111c810dba1e861fbcc71a1bf8d920e479448c However this ODE alone cannot fully describe h(t) uniquely. You also need an initial condition. Mathematically this is because to solve a first-order differential equation in time, you need 1 initial condition. Logically, to know height as a function of time, you need to know the starting height from which the ball was thrown. Look at the image below. In both cases, the balls are thrown at the exact same time with the exact same initial velocity component in the vertical direction. But the h(t) depends on the initial height. So you need to know h(t=0) for fully describing the height of the ball as a function of time. https://preview.redd.it/eobv9u1mpwdg1.png?width=1100&format=png&auto=webp&s=a28a6c8584f37683f703b4c72a5a8f436353dedc This means it is not enough to make the neural network make accurate predictions on dh/dt, the neural network should also make accurate prediction on h(t=0) for fully matching the physics in this case. # Loss due to dh/dt (ODE loss) We know the expected dh/dt because we know the initial velocity and acceleration due to gravity. How do we get the dh/dt predicted by the neural network? After all it is predicting height h, not velocity v or dh/dt. The answer is **Automatic differentiation (AD).** Because most machine‐learning frameworks (e.g., TensorFlow, PyTorch, JAX) support automatic differentiation, you can compute dh/dt by differentiating the neural network. Thus, we have a predicted dh/dt (from the neural network differentiation) for every experimental time points, and we have an actual dh/dt based on the physics. https://preview.redd.it/msf6gyunpwdg1.png?width=1100&format=png&auto=webp&s=1392d9e60f5ee011a480392af07e05bc5d094492 Now we can define a loss due to the difference between predicted and physics-based dh/dt. https://preview.redd.it/68xl4xpopwdg1.png?width=1100&format=png&auto=webp&s=5b9a727be489bd8736e8ffc235f49fca5dc25b9a Minimizing this loss (which I prefer to call ODE loss) is a good thing to ensure that neural network learns the ODE. But that is not enough. We need to make the neural network follow the initial condition also. That brings us to the next loss term.Initial condition loss # Initial condition loss This is easy. You know the initial condition. You make the neural network make a prediction of height for t=0. See how far off the prediction is from the reality. You can construct a squared error which can be called as the *Initial Condition Loss.* https://preview.redd.it/4u4syj1qpwdg1.png?width=1100&format=png&auto=webp&s=591b7e0f46ebf32024533c9d727042a889c3007d So is that it? You have ODE loss and Initial condition loss. Is it enough that the neural network tries to minimize these 2 losses? What about the experimental data? There are 3 things to consider. 1. You cannot throw away the experimental data. 2. You cannot neglect the physics described by the ODEs or PDEs. 3. You cannot neglect the initial and/or boundary conditions. Thus you have to also consider the data-based mean squared error loss along with ODE loss and Initial condition loss. # The modified loss term The simple mean squared error based loss term can now be modified like below. https://preview.redd.it/n2xc18prpwdg1.png?width=1100&format=png&auto=webp&s=95fabc8b54b2b291292d6ab2c15f5810c13379ce If there are boundary conditions in addition to initial conditions, you can add an additional term based on the difference between predicted boundary conditions and actual boundary conditions. https://preview.redd.it/ezh3in7spwdg1.png?width=1100&format=png&auto=webp&s=70367e6fbb1aa6e7924d93da8ff3b0ce8898419d Here the Data loss term ensures that the predictions are not too far from the experimental data points. The *ODE loss term* \+ the *initial condition loss term* ensures that the predictions are not too far from what described by the physics. If you are pretty sure about the physics the you can set λ1 to zero. In the ball throwing experiment, you will be sure about the physics described by our ODE if air drag, wind, buoyancy and any other factors are ignored. Only gravity is present. And in such cases, the PINN effectively becomes an ODE solver. However, for real life cases where only part of the physics is known or if you are not fully sure of the ODE, then you retain λ1 and other λ terms in the net loss term. That way you force the neural network to respect physics as well as the experimental data. This also suppress the effects of experimental noise and outliers. >