PhyNet: Physics Guided Neural Networks for Particle Drag ...people.cs.vt.edu/naren/papers/PhyNet_SDM_2020.pdfPhysics-guided design of neural network architectures, (2) Learning with

PhyNet: Physics Guided Neural Networksfor Particle Drag Force Prediction in Assembly

Nikhil Muralidhar ∗† Jie Bu ∗† Ze Cao ‡ Long He ‡ Naren Ramakrishnan ∗

Danesh Tafti ‡ Anuj Karpatne ∗†

Abstract

Physics-based simulations are often used to model and un-

derstand complex physical systems in domains like fluid dy-

namics. Such simulations although used frequently, often

suffer from inaccurate or incomplete representations either

due to their high computational costs or due to lack of com-

plete physical knowledge of the system. In such situations,

it is useful to employ machine learning to fill the gap by

learning a model of the complex physical process directly

from simulation data. However, as data generation through

simulations is costly, we need to develop models being cog-

nizant of data paucity issues. In such scenarios it is helpful

if the rich physical knowledge of the application domain is

incorporated in the architectural design of machine learn-

ing models. We can also use information from physics-based

simulations to guide the learning process using aggregate su-

pervision to favorably constrain the learning process. In

this paper, we propose PhyNet , a deep learning model using

physics-guided structural priors and physics-guided aggregate

supervision for modeling the drag forces acting on each par-

ticle in a Computational Fluid Dynamics-Discrete Element

Method (CFD-DEM). We conduct extensive experiments in

the context of drag force prediction and showcase the use-

fulness of including physics knowledge in our deep learning

formulation. PhyNet has been compared to several state-of-

the-art models and achieves a significant performance im-

provement of 8.46% on average . The source code has been

made available∗ and the dataset used is detailed in [1, 2].

1 Introduction

Machine learning (ML) is ubiquitous in several disci-plines today and with its growing reach, learning mod-els are continuously exposed to new challenges andparadigms. In many applications, ML models aretreated as black-boxes. In such contexts, the learningmodel is trained in a manner completely agnostic to therich corpus of physical knowledge underlying the process

∗Dept. of Computer Science, Virginia Tech, VA, USA†Discovery Analytics Center, Virginia Tech, VA, USA‡Dept. of Mechanical Engineering, Virginia Tech, VA, USA∗https://github.com/nmuralid1/PhyNet.git

PhyNet

Physics-Guided Model ArchitectureInputs(Reynolds Number,

Solid Fraction, Neighboring

Particle Positions)

Physical Loss

(Aggregate Supervision)

OutputsDrag Force !"

Pressure FieldsVelocity Fields

Intermediate Variables

Figure 1: Our proposed PhyNet Model.

being modeled. This domain-agnostic training mightlead to many unintended consequences like the modellearning spurious relationships between input variables,models learning representations that are not easily ver-ifiable as being consistent with the accepted physicalunderstanding of the process being modeled. Moreover,in many scientific disciplines, generating training datamight be extremely costly due to the nature of the datageneration, collection process. To be used across manyscientific applications, it is important for data miningmodels to leverage the rich physical knowledge in sci-entific disciplines to fill the void due to data paucityand be able to learn good process representations in thecontext of limited data. This makes the model less ex-pensive to train as well as more interpretable due tothe ability to verify whether the learned representationis consistent with existing domain knowledge. In thispaper, we attempt to bridge the gap between physics-based models and data mining models by incorporatingdomain knowledge in the design and learning of machinelearning models. Specifically, we propose three ways forincorporating domain knowledge in neural networks: (1)Physics-guided design of neural network architectures,(2) Learning with auxiliary tasks involving physical in-termediate variables, and (3) Physics-guided aggregatesupervision of neural network training.

We focus on modeling a system in the domain ofmulti-phase flows (solid particles suspended in movingfluid) which have a wide range of applicability in fun-damental as well as industrial processes [3]. One of thecritical interaction forces in these systems that has a

Copyright © 2020 by SIAMUnauthorized reproduction of this article is prohibited

large bearing on the dynamics of the system is the dragforce applied by the fluid on the particles and vice-versa.The drag force can be obtained by Particle ResolvedSimulations (PRS) at a high accuracy. It captures thevelocity and pressure field surrounding each particle inthe suspension that can later be used to compute thedrag force. However, PRS is quite expensive and only afew 100s or at most 1000s of particles can be resolved ina calculation utilizing grids of O(108) degrees of freedomand utilizing O(102) processors or cores. Thus morepractical simulations have to resort to coarse-graining,e.g., Discrete Element Method (DEM) and CFD-DEM.In these methods, a particle is treated as a point mass(not resolved) and the fluid drag force acting on theparticles in suspension is modeled.

Current practice in simulations is to use the meandrag force acting on the particle suspension as a functionof flow parameters (Reynolds number) and the particlepacking density (solid fraction - φ) [4–6]. Giventhe variability of drag force on individual particles insuspension, this paper explores techniques in physics-guided machine learning to advance the current state-of-the-art for drag force prediction in CFD-DEM bylearning from a small amount of PRS data.Our contributions are as follows:• We introduce PhyNet , a novel physics-guided modelarchitecture that yields state-of-the-art results for theproblem of particle drag force prediction.• We introduce physics-guided auxiliary tasks to trainPhyNet more effectively with limited data.• We augment PhyNet architecture with aggregate su-pervision applied over the auxiliary tasks to ensure con-sistency with physics knowledge.• Finally, we conduct extensive experimentation to un-cover several useful properties of our model in settingswith limited data and showcase that PhyNet is consis-tent with existing physics knowledge about factors in-fluencing drag force over a particle, thus yielding greatermodel interpretability.

2 Related Work

There have been multiple efforts to leverage domainknowledge in the context of increasing the performanceof data-driven or statistical models. Methods havebeen designed to influence training algorithms in MLusing domain knowledge, e.g., with the help of phys-ically based priors in probabilistic frameworks [7–9],regularization terms in statistical models [10, 11], con-straints in optimization methods [12, 13], and rules inexpert systems [14, 15]. In a recent line of research,new types of deep learning models have been proposed(e.g., ODEnet [16] and RKnet [17]) by treating sequen-tial deep learning models such as residual networks and

recurrent neural networks as discrete approximations ofordinary differential equations (ODEs).

Yaser et al. show hints, i.e., prior knowledge can beincorporated into learning-from-example paradigm [15].In [18] the authors explored the idea of incorporatingdomain knowledge directly as a regularizer in neuralnetworks to influence training and showed better gener-alization performance. In [19,20] domain knowledge wasincorporated into a customized loss function for weaksupervision that relies on no training labels.

There have been efforts to incorporate prior knowl-edge about a problem (like low rank structure of con-volutional filters to be designed) into model architec-ture design (structural priors) [21]. Also, to design neu-ral network architectures, to incorporate feature invari-ance [22], implicit physics rules [23] to enable learningrepresentations consistent with physics laws and explic-itly incorporating knowledge as constraints [24]. In [25]the authors propose a neural network model where eachneuron learns ”laws” similar to physics laws applied tolearn the behavior of complex many-body physical sys-tems. In [26], the authors propose a theory that detailshow to design neural network architectures for data withnon-trivial symmetries. However none of these effortsare directly applicable to encode the physical relation-ships we are interested in modeling.

3 Proposed PhyNet Framework

3.1 Problem Background: Given a collection of N3D particles suspended in a fluid moving along the Xdirection, we are interested in predicting the drag forceexperienced by the ith particle, Fi, along theX directiondue to the moving fluid. This can be treated as asupervised regression problem where the output variableis Fi, and the input variables include features capturingthe spatial arrangement of particles neighboring particlei, as well as other attributes of the system such asReynolds Number, Re, and Solid Fraction (fraction ofunit volume occupied by particles), φ. Specifically,we consider the list of 3D coordinates of 15-nearestneighbors around particle i, appended with (Re, φ) asthe set of input features, represented as a flat 47-lengthvector, Ai.

A simple way to learn the mapping from Ai to Fiis by training feed-forward deep neural network (DNN)models, that can express highly non-linear relationshipsbetween inputs and outputs in terms of a hierarchy ofcomplex features learned at the hidden layers of thenetwork. However, black-box architectures of DNNswith arbitrary design considerations (e.g., layout of thehidden layers) can fail to learn generalizable patternsfrom data, especially when training size is small. Toaddress the limitations of black-box models in our target


[1]

Input Layer [1]Fully-connected LayerAcvaon: ELUInput dimension: 47Output dimension: 128

[2]

Shared Layer(s) [2]Fully-connected LayerAcvaon: ELUInput dimension: 128Output dimension: 128

Pressure Field [3]Fully-connected LayerAcvaon: ELUInput dimension: 128Output dimension: 10

[3]

Velocity Field [4]Fully-connected LayerAcvaon: ELUInput dimension: 128Output dimension: 10

[4]

[5]

Convoluon Layer [5]1D Convoluonal LayerAcvaon: LinearInput dimension: (2,10)Output dimension: (2, 8)

[6]

Pooling Layer [6]1D MaxPooling LayerAcvaon: LineaerInput dimension: (2, 8)Output dimension: (2, 4)

[7]

Shear Component [7]Fully-connected LayerAcvaon: LinearInput dimension: 4Output dimension: 3

[8]Pressure Component [8]Fully-connected LayerAcvaon: LinearInput dimension: 4Output dimension: 3

[9]

Output Layer [9]Fully-connected LayerAcvaon: LinearInput dimension: 6Output dimension: 1

Figure 2: PhyNet Architecture

application of drag force prediction, we present a novelphysics-guided DNN model, termed PhyNet , that usesphysical knowledge in the design and learning of theneural network, as described in the following.

3.2 Physics-guided Model Architecture: In or-der to design the architecture of PhyNet , we derive in-spiration from the known physical pathway from theinput features Ai to drag force Fi, which is at the ba-sis of physics-based model simulations such as ParticleResolved Simulations (PRS). Essentially, the drag forceon a particle i can be easily determined if we know twokey physical intermediate variables: the pressure field(Pi) and the velocity field (Vi) around the surface ofthe particle. It is further known that Pi directly affectsthe pressure component of the drag force, FPi , and Vi

directly affects the shear component of the drag force,FSi . Together, FPi and FSi add up to the total dragforce that we want to estimate, i.e., Fi = FPi + FSi .

Using this physical knowledge, we design ourPhyNet model so as to express physically meaningful in-termediate variables such as the pressure field, velocityfield, pressure component, and shear component in theneural pathway from Ai to Fi. Figure 2 shows the com-plete architecture of our proposed PhyNet model withdetails on the number of layers, choice of activationfunction, and input and output dimensions of everyblock of layers. In this architecture, the input layerpasses on the 47-length feature vectors Ai to a collec-tion of four Shared Layers that produce a common setof hidden features to be used in subsequent branchesof the neural network. These features are transmittedto two separate branches: the Pressure Field Layer andthe Velocity Field Layer, that express Pi and Vi, respec-tively, as 10-dimensional vectors. Note that Pi and Vi

represent physically meaningful intermediate variablesobserved on a sequence of 10 equally spaced points onthe surface of the particle along the X direction.

The outputs of pressure field and velocity fieldlayers are combined and fed into a 1D Convolutionallayer that extracts the sequential information containedin the 10-dimensional Pi and Vi vectors, followedby a Pooling layer to produce 4-dimensional hiddenfeatures. These features are then fed into two newbranches, the Shear Component Layer and the PressureComponent Layer, expressing 3-dimensional FS

i and FPi ,

respectively. These physically meaningful intermediatevariables are passed on into the final output layer thatcomputes our target variable of interest: drag forcealong the X direction, Fi. Note that we only make useof linear activation functions in all of the layers of ourPhyNet model following the Pressure Field and VelocityField layers. This is because of the domain informationthat once we have extracted the pressure and velocityfields around the surface of the particle, computing Fi isrelatively straightforward. Hence, we have designed ourPhyNet model in such a way that most of the complexityin the relationship from Ai to Fi is captured in thefirst few layers of the neural network. The layout ofhidden layers and the connections among the layers inour PhyNet model is thus physics-guided. Further, thephysics-guided design of PhyNet ensures that we hingesome of the hidden layers of the network to expressphysically meaningful quantities rather than arbitrarilycomplex compositions of input features, thus adding tothe interpretability of the hidden layers.

3.3 Learning with Physical Intermediates: It isworth mentioning that all of the intermediate variablesinvolved in our PhyNet model, namely the pressure fieldPi, velocity field Vi, pressure component FP

i , and shearcomponent FS

i , are produced as by-products of thePRS simulations that we have access to during training.Hence, rather than simply learning on paired examplesof inputs and outputs, (Ai, Fi), we consider learning ourPhyNet model over a richer representation of training


examples involving all intermediate variables along withinputs and outputs. Specifically, for a given input Ai,we not only focus on accurately predicting the outputvariable Fi at the output layer, but doing so whilealso accurately expressing every one of the intermediatevariables (Pi,Vi,F

Pi ,F

Si ) at their corresponding hidden

layers. This can be achieved by minimizing the followingempirical loss during training:

LossMSE = λP MSE(P, P̂) + λVMSE(V, V̂) +

λFPMSE(FP, F̂P) + λFSMSE(FS, F̂S) +MSE(F, F̂ )

where MSE represents the mean squared error, x̂ repre-sents the estimate of x, and λP , λV , λFP , and λFS rep-resent the trade-off parameters in miniming the errorson the intermediate variables. Minimizing the aboveequation will help in constraining our PhyNet modelwith loss terms observed not only on the output layerbut also on the hidden layers, grounding our neural net-work to a physically consistent (and hence, generaliz-able) solution. Note that this formulation can be viewedas a multi-task learning problem, where the predictionof the output variable can be considered as the primarytask, and the prediction of intermediate variables can beviewed as auxiliary tasks that are related to the primarytask through physics-informed connections, as capturedin the design of our PhyNet model.

3.4 Using Physics-guided Loss: Along with learn-ing our PhyNet using the empirical loss observed ontraining samples, LossMSE , we also consider adding anadditional loss term that captures our physical knowl-edge of the problem and ensures that the predictionsof our PhyNet model do not violate known physicalconstraints. In particular, we know that the distribu-tion of pressure and velocity fields over different com-binations of Reynolds number (Re) and solid fraction(φ) show varying aggregate properties (e.g., differentmeans), thus exhibiting a multi-modal distribution. Ifwe train our PhyNet model on data instances belong-ing to all (Re,φ) combinations using LossMSE , we willobserve that the trained model will under-perform onsome of the modes of the distribution that are under-represented in the training set. To address this, wemake use of a simple form of physics-guided aggregatesupervision, where we enforce the predictions P̂(Re,φ)

and V̂(Re,φ) of the pressure and velocity fields arounda particle respectively, at a given combination of (Re,φ)to be close to the mean of the actual values of P and Vproduced by the PRS simulations at that combination.If P (Re,φ) and V (Re,φ) represent the mean of the pres-sure and velocity field respectively for the combination(Re, φ), we consider minimizing the following physics-

guided loss:

LossPHY =∑Re

∑φ

MSE(µ(P̂(Re,φ)), P (Re,φ))

+MSE(µ(V̂(Re,φ)), V (Re,φ))

The function µ(·) : R −→ R is a mean function. Wefinally consider the combined loss LossMSE +LossPHYfor learning our PhyNet model.

4 Dataset Description

The dataset used has 5824 particles. Each particle has47 input features including three-dimensional coordi-nates for fifteen nearest neighbors relative to the targetparticle’s position, the Reynolds number (Re) and solidfraction (φ) of the specific experimental setting (thereare a total of 16 experimental settings with different(Re, φ) combinations). Labels include the drag force inthe X-direction Fi ∈ R1×1 as well as variables for auxil-iary training, i.e., pressure fields (Pi ∈ R10×1), velocityfields (Vi ∈ R10×1), pressure components (FP

i ∈ R3×1)and shear components of the drag force (FS

i ∈ R3×1)†.

4.1 Experimental Setup All deep learning modelsused have 5 hidden layers, a hidden size of 128 andwere trained for 500 epochs with a batch size of 100.Unless otherwise stated, 55% of the dataset was used fortraining while the remaining data was used for testingand evaluation. We applied standardization to the allinput features and labels in the data preprocessing step.

Baselines: We compare the performance ofPhyNet with several state-of-the-art regression baselinesand a few close variants of PhyNet .•Linear Regression (Linear Reg.), Random Forest Re-gression (RF Reg.), Gradient Boosting Regression (GBReg.) [27]: We employed an ensemble of 100 estimatorsfor RF, GB Reg. models and left all other parametersunchanged.• DNN: A standard feed-forward neural network modelfor predicting the scalar valued particle drag force Fi.• DNN+ Pres: A DNN model which predicts the pres-sure field around a particle (Pi) in addition to Fi.• DNN+ Vel: A DNN model which predicts the velocityfield around a particle (Vi) in addition to Fi.• DNN-MT-Pres: Similar to DNN+ Pres except thatthe pressure and drag force tasks are modeled in a multi-task manner with a set of disjoint layers for each of thetwo tasks and a separate set of shared layers.•DNN-MT-Vel; Similar to DNN-MT-Pres except in thiscase the auxiliary task models the velocity field aroundthe particle (Vi) in addition to drag force (Fi).

†Further details about the dataset included in the appendix -https://bit.ly/2RDwOBa .


We employ three metrics for model evaluation:MSE & MRE: We employ the mean squared error

(MSE) and mean relative error (MRE) [2] metrics toevaluate model performance. Though MSE can capturethe absolute deviation of model prediction from theground truth values, it can vary a lot for different scalesof the label values, e.g., for higher drag force values,MSE is prone to be higher, vice versa. Thus, the needfor a metric that is invariant to the scale of the labelvalues brings in the MRE as an important supplementalmetric in addition to MSE.

MRE =1

m

m∑i=1

|F̂i − Fi|F (Re,φ)

F (Re,φ) is the mean drag force for (Re, φ) set-

ting and F̂i the predicted drag force for particle i.

0.00.0

0.2

0.4

0.6

0.8

0.2 0.4 0.6 0.8 1.0

1.0Example Relative Error (RE) Curve

Relative Error (RE)

Cummulative Distribution

Sample Non-Ideal RE-Curve

Ideal RE-Curve

AU-REC = 1.0

AU-REC < 1.0

AU-REC: The thirdmetric we employ is thearea under the relativeerror curve (AU-REC).The relative error curverepresents the cumu-lative distribution ofrelative error betweenthe predicted drag forcevalues and the groundtruth PRS drag forcedata. AU-REC calcu-lates the area under thiscurve. The AU-REC metric ranges between [0,1] andhigher AU-REC values indicate superior performance.

5 Experimental Results

We conducted multiple experiments to characterizeand evaluate the model performance of PhyNet withphysics-guided architecture and physics-guided aggregatesupervision. Cognizant of the cost of generation of dragforce data, we aim to evaluate models in settings wherethere is a paucity of labelled training data.

5.1 Physics-Guided Auxiliary Task SelectionWhen data about the target task is limited, we may em-ploy exogenous inputs of processes that have an indirectinfluence over the target process to alleviate the effectsof data paucity on model training. An effective wayto achieve this is through multi-task learning. We firstevaluate multi-task model performance relative to thecorresponding single-task models to demonstrate per-formance gains. Table 1 shows the results of severalmulti-task and single task architectures that we testedto establish the superiority of multi-task models in the

Model MSE MRE AU-REC(% Imp.)

Linear Reg. 47.47 38.48 0.71332 (-19.54)

RF Reg. 29.33 19.13 0.82148 (-7.3)

GB Reg. 25.02 17.55 0.83692 (-5.60)

DNN 20.50 16.72 0.84573 (-4.61)

DNN-MT-Pres 20.12 15.66 0.85593 (-3.45)

DNN-MT-Vel 19.98 15.69 0.85556 (-3.49)

PhyNet-FPx FS

x 15.54 14.06 0.87232 (-1.61)

PhyNet 14.28 12.59 0.88657 (–)

Table 1: We compare the performance of PhyNet andits variant PhyNet-FP

x FSx (only x-components of pressure

and shear drag are modeled) with many state-of-the-artregression baselines and show that the PhyNet model yieldssignificant performance improvement over all other modelsfor the particle drag force prediction task. The last columnof the table reports the AU-REC metric and also quantifiesthe percentage improvement of the best performing model i.ePhyNet w.r.t all other models in the context of the AU-RECmetric.

context of the particle drag force prediction task. Itis widely known and accepted in physics that the dragforce on each particle in fluid-particle systems such asthe one being considered in this paper, is influencedstrongly by the pressure and velocity fields acting onthe particles [2]. Hecnce, we wish to explicitly model thepressure and velocity fields around a particle, in addi-tion to the main problem of predicting its drag force. Tothis end, we design two multi-task models, DNN-MT-Pres, DNN-MT-Vel, as described in section 4.1. Wenotice that the two multi-task models DNN-MT-Presand DNN-MT-Vel outperform not only the DNN modelbut also their single task counterparts (DNN+ Pres ,DNN+ Vel). This improvement in performance may beattribured to the carefully selected auxiliary tasks to aidin learning the representation of the main task. Thisphysics-guided auxiliary task selection is also impera-tivie to our process of development of PhyNet modelsto be detailed in section 5.2. However, Table 1 also un-covers another interesting property which is the DNN+models underperforming compared to their DNN-MTcounterparts. Although the DNN+ and the DNN-MTmodels are predicting the same set of 11 values i.e 1drag force value and 10 pressure or velocity samples inthe vicinity of the particle, the DNN+ models maketheir predictions as part of a single task. Hence, theimportance of the main task is diminished by the 10additional auxiliary task values as the model tries tolearn a jointly optimal representation. However, in thecase of the DNN-MT models, each task has a set of dis-joint hidden layers geared specifically towards learningthe representation of the main task and a similar setof layers for the auxiliary task (in addition to a set of


shared layers), which yields more flexibility in learningrepresentations specific to the main and auxiliary taskas well as a shared common representation. In addi-tion, it is straightforward to explicitly control the effectof auxiliary tasks on the overall learning process in amulti-task setting.

5.2 Physics-Guided Learning Architecture Sec-tion 5.1 showcases the effectiveness of multi-task learn-ing and of physics-guided auxiliary task selection forlearning improved representations of particle drag force.We now delve deeper and inspect the effects of expand-ing the realm of auxiliary tasks. In addition to this, wealso use our domain knowledge regarding the physicsof entities affecting the drag force acting on each par-ticle, to influence model architecture through physics-guided structural priors. As mentioned in Section 3,PhyNet has four carefully and deliberately chosen aux-iliary tasks (pressure field prediction, velocity field pre-diction, predicting the pressure component(s) of drag,predicting the shear components of drag) aiding themain task of particle drag force prediction. In additionto this, the auxiliary tasks are arranged in a sequen-tial manner to incorporate physical inter-dependenciesamong them leading up to the main task of parti-cle drag force prediction. The effect of this carefullychosen physics-guided architecture and auxiliary taskscan be observed in Table 1. We now inspect the dif-ferent facets of this physics-guided architecture of thePhyNet model‡.

0.0 0.2 0.4 0.6 0.8 1.0Percentage Error Compared to PRS Data

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ula

tive D

istr

ibuti

on

DNN

PhyNet

Mean

Figure 3: The cumulative distribution function of relative er-ror for all (Re,φ) combinations. Overall, the PhyNet modelcomfortably outperforms the DNN model and the Meanbaseline (dotted red line).

We first characterize the performance of ourPhyNet models with respect to the DNN and meanbaseline. Fig. 3 represents the cumulative distribution

‡PhyNetwas found to be robust to changes in auxiliary taskhyperparameters, results in appendix - https://bit.ly/2RDwOBa.

of relative error of the predicted drag forces and thePRS ground truth drag force data. We notice that bothDNN and PhyNet outperform the mean baseline whichessentially predicts the mean value per (Re,φ) combi-nation. The PhyNet model significantly outperformsthe DNN (current state-of-the-art [2]) model to yieldthe best performance overall. We also tested DNNvariants with dropout and L2 regularization but foundthat performance deteriorated. Results were excludeddue to space constraints.

5.3 Performance With Limited Data Bearing inmind the high data generation cost of the PRS simu-lation, we wish to characterize an important facet ofthe PhyNet model, namely, its ability to learn effec-tive representations when faced with a paucity of train-ing data. Hence, we evaluate the performance of thePhyNet model as well as the other single task and multi-task DNN models, on different experimental settingsobtained by continually reducing the fraction of dataavailable for training the models. In our experiments,the training fraction was reduced from 0.85 (i.e 85% ofthe data used for training) to 0.35 (i.e 35% of the dataused for training).

0.35 0.45 0.55 0.65 0.75 0.85Training Fraction

0.75

0.80

0.85

0.90

AU

-REC

PhyNet

RF Reg.

Linear Reg.

GB Reg.

Mean

DNN

Figure 4: Model performance comparison for different levelsof data paucity.

Fig. 4 showcases the model performance in settings withlimited data. We see that PhyNet model significantlyoutperforms all other models in most settings (sparseand dense). We note that GB Reg. yields comparableperformance to the PhyNet model for the case of 0.35training fraction. However, the gradient boosting (andall the other regression models except DNN) fail to learnuseful information as more data is provided for training.We also notice that the DNN model fails to outperformthe PhyNet model for all but the last setting i.e thesetting with 0.85 training fraction.

5.4 Characterizing PhyNet Performance ForDifferent (Re,φ) Settings. In addition to quanti-tative evaluation, qualitative inspection is necessary


for a deeper, holistic understanding of model behav-ior. Hence, we showcase the particle drag force pre-dictions by the PhyNet model for different (Re,φ) com-binations in Fig. 5§. We notice that the PhyNet modelyields accurate predictions (i.e yellow and red curvesare aligned). This indicates that the PhyNet model isable to effectively capture sophisticated particle inter-actions and the consequent effect of said interactions onthe drag forces of the interacting particles. We noticethat for high (Re,φ) as in Fig. 5b, the drag force i.ePRS curve (yellow) is nonlinear in nature and that themagnitude of drag forces is also higher at higher (Re,φ)settings. Such differing scales of drag force values canalso complicate the drag force prediction problem as itis non-trivial for a single model to effectively learn suchmulti-modal target distributions. However, we find thatthe PhyNet model is effective in this setting.

0 20 40 60 80 100 120 140Particle Index

2

4

6

8

10

12

Dra

g F

orc

e

Mean

PRS

PhyNet Avg.

PhyNet

(a) Re = 10, φ = 0.2

0 50 100 150 200Particle Index

40

60

80

100

Dra

g F

orc

e

Mean

PRS

PhyNet Avg.

PhyNet

(b) Re = 200, φ = 0.35

Figure 5: Each figure shows a comparison betweenPhyNet predictions (red curve) and ground truth drag forcedata (yellow curve), for different (Re,φ) cases. We also show-case the mean drag force value for each (Re,φ) case (black).

Thus far, we characterized the performance of thePhyNet model in isolation for different (Re,φ) contexts.In order to gain a deeper understanding of the perfor-mance of PhyNet models for different (Re,φ) combina-tions, we show percentage improvement for the AU-REC metric of PhyNet model and three other mod-els in Fig. 6a - Fig. 6c. We choose DNN, DNN-MT-Pres, DNN-MT-Vel as these are the closest by design toPhyNet among all the baselines we consider in this pa-per. In Fig. 6, we see that PhyNet outperforms the othermodels in most of the (Re,φ) settings. PhyNet whencompared with the DNN model achieves especially goodperformance for low solid fraction settings which may beattributed to the inability of the DNN model to learneffectively with low data volumes as lower solid fractionshave fewer training instances. In the case of the DNN-MT models, the PhyNet model achieves significant per-formance improvement for high solid fraction (φ = {0.2,0.3, 0.35}) cases and also for Re = {10, 200}, indicating

§Figures for all (Re,φ) combinations are in the appendix.

that PhyNet is able to perform well in the most com-plicated scenarios (high Re, high φ). PhyNet is able toachieve superior performance in 14 out of the 16 (Re,φ) settings across all three models.

5.5 Verifying Consistency With DomainKnowledge A significant advantage of physics-guidedmulti-task structural priors is the increased inter-pretability provided by the resulting architecture.Since each component of the PhyNet model has beendesigned and included based on sound domain theory,we may employ this theoretical understanding to verifythrough experimentation that the resulting behaviorof each auxiliary component is indeed consistent withknown theory. We first verify the performance of thepressure and shear drag component prediction task inthe PhyNet model. It is well accepted in theory thatfor high Reynolds numbers, the proportion of the shearcomponents of drag (FS) decreases [2]. In order toevaluate this, we consider the ratio of the magnitude ofthe predicted pressure components in the x-direction(FPx ∈ FP) to the magnitude of the predicted shearcomponents in the x-direction (FSx ∈ FS) for every (Re,φ) setting¶. The heatmap in Fig. 7 depicts the compar-ison of this ratio of predicted pressure components topredicted shear components to a similar ratio derivedfrom the ground truth pressure and shear components.We notice that there is good agreement between thepredicted and ground truth ratios for each (Re, φ)setting and also that the behavior of the predictedsetting is indeed consistent with known domain theoryas there is a noticeable decrease in the contribution ofthe shear components as we move toward high Re andhigh solid fraction φ settings.

5.6 Auxiliary Representation Learning WithPhysics-Guided Statistical Constraints Two ofthe auxiliary prediction tasks involve predicting thepressure and velocity field samples around each particle.We hypothesized that since the drag force of a particleis influenced by the pressure and velocity fields, model-ing them explicitly should help the model learn an im-proved representation of the main task of particle dragforce prediction. In Fig. 8, we notice that ground-truthpressure field PDFs exhibit a grouped structure. In-terestingly, the pressure field PDFs can be divided intothree distinct groups with all the pressure fields withφ = 0.2 being grouped to the left of the plot, pressurefields with φ = 0.1 being grouped toward the bottom,right of the plot and the rest of the PDFs forming a core

¶Similar behavior was recorded even when ratios were takenfor all three pressure and shear drag components.


(a) PhyNet vs. DNN (b) PhyNet vs. DNN-MT-Pres. (c) PhyNet vs. DNN-MT-Vel

Figure 6: Each figure indicates the percentage improvement in the context of the AU-REC metric of the PhyNet modelover the DNN (Fig. 6a), DNN-MT-Pres (Fig. 6b) and DNN-MT-Vel (Fig. 6c). Red squares show that PhyNet does betterand blue squares indicate that other models outperform PhyNet . PhyNet yields significant performance improvement overother models.

0.1 0.2 0.3 0.35Solid Fraction

1050

100

200

Reyn

olds

Num

ber

0.83 1.00 1.30 1.68

0.63 1.21 1.52 2.15

0.74 1.66 1.93 2.77

1.47 2.43 3.05 3.65

Predictions

0.1 0.2 0.3 0.35Solid Fraction

1050

100

200

Reyn

olds

Num

ber

0.82 1.15 1.45 1.78

1.02 1.40 1.72 2.18

1.24 1.65 2.04 2.58

1.68 2.36 2.95 3.61

Ground Truth

1.0

1.5

2.0

2.5

3.0

3.5

1.0

1.5

2.0

2.5

3.0

3.5

Figure 7: Heatmap with ratio of absolute value of pressuredrag (FP

x ) x-component to shear drag (FSx ) x-component

i.e( |FP

x ||FS

x |

). Left figure shows ratio for PhyNet predictions

and right figure shows the same ratio for ground truth data.Distribution of ratios in both figures is almost identical.

(highly dense) group in the center. Hence, we infer thatsolid fraction has a significant influence on the pressurefield. It is non-trivial for models to automatically repli-cate such multi-modal and grouped behavior and hencewe introduce physics-guided statistical priors throughaggregate supervision during model training of PhyNet .We notice that the learned distribution with aggregatesupervision Fig. 8 (center) has a similar grouped struc-ture to the ground truth PDF pressure field. We alsoobtained the predicted pressure field PDFs of a versionof PhyNet trained without aggregate supervision andthe result is depicted in Fig. 8 (right). We notice thatthe PDFs exhibit a kind of mode collapse behavior anddo not display any similarities to ground truth pressurefield PDFs. Similar aggregate supervision was also ap-plied to the velocity field prediction task and we foundthat incorporating physics-guided aggregate supervisionto ensure learning representations consistent with the-ory, significantly improved model performance.

6 Conclusion

In this paper, we introduce PhyNet , a physics inspireddeep learning model developed to incorporate fluid me-chanical theory into the model architecture and pro-posed physics informed auxiliary tasks selection to aidwith training under data paucity. We conduct a rigor-ous analysis to test PhyNet performance in settings withlimited labelled data and find that PhyNet significantlyoutperforms all state-of-the-art baselines for the taskof particle drag force prediction, achieving an averageperformance improvement of 8.46%. We verify thateach physics informed auxiliary task of PhyNet is con-sistent with existing physics theory, yielding greatermodel interpretability. Finally, we showcase the effect ofaugmenting PhyNet with physics-guided aggregate su-pervision to constrain auxiliary tasks to be consistentwith ground truth data. In future, we will augmentPhyNet with more sophisticated learning architectures.

7 Acknowledgements

This work is supported by the National Science Foun-dation via grant DGE-1545362.

References

[1] L. He, D. K. Tafti, and K. Nagendra, “Evaluation ofdrag correlations using particle resolved simulations ofspheres and ellipsoids in assembly,” Powder technology,vol. 313, 2017.

[2] L. He and D. K. Tafti, “A supervised machine learningapproach for predicting variable drag forces on spher-ical particles in suspension,” Powder technology, vol.345, 2019.

[3] J. Li and J. Kuipers, “Gas-particle interactions indense gas-fluidized beds,” Chemical Engineering Sci-ence, vol. 58, no. 3-6, 2003.


1 0 1 2 3 4 5Ground Truth

0.0

0.5

1.0

1.5

2.0

2.5

Re=10-SF=0.1Re=10-SF=0.2Re=10-SF=0.3Re=10-SF=0.35




6 4 2 0 2 4 6Prediction With Aggregate Supervision

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

6 4 2 0 2 4 6Prediction Without Aggregate Supervision

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Figure 8: The figure depicts the densities of the ground truth (left) and predicted (center, right) pressure fields of thePhyNet model for each (Re,φ). Specifically, we wish to highlight the effect of aggregate supervision (physics-guidedstatistical prior) on the predicted pressure field. Notice that the PDFs of the pressure fields predicted with aggregatesupervision are relatively more distributed similar to the ground truth distribution of pressure field PDFs as opposed tothe plot on the right which represents predicted pressure field PDFs in the abscence of aggregate supervision and incorrectlydepicts a some what uniform behavior for all the PDFs of different (Re,φ) cases.

[4] C. Y. Wen, “Mechanics of fluidization,” in ChemicalEngineering Progress Symposium Series, vol. 62, 1966,pp. 100–111.

[5] R. Di Felice, “The voidage function for fluid-particleinteraction systems,” International Journal of Multi-phase Flow, vol. 20, no. 1, 1994.

[6] S. Tenneti, R. Garg, and S. Subramaniam, “Draglaw for monodisperse gas–solid systems using particle-resolved direct numerical simulation of flow past fixedassemblies of spheres,” International journal of multi-phase flow, vol. 37, no. 9, 2011.

[7] K. C. Wong, L. Wang, and P. Shi, “Active model withorthotropic hyperelastic material for cardiac imageanalysis,” in Functional Imaging and Modeling of theHeart. Springer, 2009.

[8] J. Xu et al., “Robust transmural electrophysiologi-cal imaging: Integrating sparse and dynamic physio-logical models into ecg-based inference,” in MICCAI.Springer, 2015.

[9] H. Denli et al., “Multi-scale graphical models forspatio-temporal processes,” in NeurIPS, 2014.

[10] S. Chatterjee et al., “Sparse group lasso: Consistencyand climate applications.” in SDM12’. SIAM, 2012.

[11] J. Liu et al., “Accounting for linkage disequilibrium ingenome-wide association studies: a penalized regres-sion method,” Statistics and its interface, vol. 6, no. 1,2013.

[12] A. J. Majda and J. Harlim, “Physics constrained non-linear regression models for time series,” Nonlinearity,vol. 26, no. 1, 2012.

[13] A. J. Majda and Y. Yuan, “Fundamental limitationsof ad hoc linear and quadratic multi-level regressionmodels for physical systems,” Discrete and ContinuousDynamical Systems B, vol. 17, no. 4, 2012.

[14] D. Waterman, “A guide to expert systems,” 1986.

[15] Y. S. Abu-Mostafa, “Learning from hints in neuralnetworks,” Journal of complexity, vol. 6, no. 2, 1990.

[16] T. Q. Chen et al., “Neural ordinary differential equa-tions,” in NeurIPS, 2018.

[17] M. Zhu, B. Chang, and C. Fu, “Convolutional neu-ral networks combined with runge-kutta methods,”arXiv:1802.08831, 2018.

[18] A. Karpatne et al., “Theory-guided data science: Anew paradigm for scientific discovery from data,” IEEETKDE, vol. 29, no. 10, 2017.

[19] H. Ren et al., “Learning with weak supervision fromphysics and data-driven constraints.” AI Magazine,vol. 39, no. 1, 2018.

[20] R. Stewart and S. Ermon, “Label-free supervision ofneural networks with physics and domain knowledge,”in AAAI, 2017.

[21] Y. A. Ioannou, “Structural priors in deep neural net-works,” Ph.D. dissertation, University of Cambridge,2018.

[22] J. Ling, A. Kurzawski, and J. Templeton, “Reynoldsaveraged turbulence modelling using deep neural net-works with embedded invariance,” Journal of FluidMechanics, vol. 807, 2016.

[23] S. Seo and Y. Liu, “Differentiable physics-informedgraph networks,” arXiv:1902.02950, 2019.

[24] J. Z. Leibo et al., “View-tolerant face recognitionand hebbian learning imply mirror-symmetric neuraltuning to head orientation,” Current Biology, 2017.

[25] B. Anderson et al., “Cormorant: Covariant molecularneural networks,” arXiv:1906.04015, 2019.

[26] R. Kondor and S. Trivedi, “On the generalization ofequivariance and convolution in neural networks to theaction of compact groups,” arXiv:1802.03690, 2018.

[27] F. Pedregosa et al., “Scikit-learn: Machine learning inPython,” JMLR, vol. 12, 2011.


PhyNet: Physics Guided Neural Networks for Particle Drag ...people.cs.vt.edu/naren/papers/PhyNet_SDM_2020.pdfPhysics-guided design of neural network architectures, (2) Learning with

Documents