人工智能论文20220917 2019PhD Cambridge The resurgence of structure in deep neural networks_PVelickovic.pdfVIP

Theresurgenceofstructure

indeepneuralnetworks

PetarVeliˇckovi´c

TrinityCollege

Thisdissertationissubmittedon15January2019

forthedegreeofDoctorofPhilosophy

Theresurgenceofstructureindeepneuralnetworks

PetarVeliˇckovi´c

Abstract

Machinelearningwithdeepneuralnetworks(“deeplearning”)allowsforlearningcom‐

plexfeaturesdirectlyfromrawinputdata,completelyeliminatinghand‐crafted,“hard‐

coded”featureextractionfromthelearningpipeline.Thishasleadtostate‐of‐the‐art

performancebeingachievedacrossseveral—previouslydisconnected—problemdomains,

includingcomputervision,naturallanguageprocessing,reinforcementlearningandgen‐

erativemodelling.Thesesuccessstoriesnearlyuniversallygohand‐in‐handwithavail‐

abilityofimmensequantitiesoflabelledtrainingexamples(“bigdata”)exhibitingsimple

grid‐likestructure(e.g.textorimages),exploitablethroughconvolutionalorrecurrent

layers.Thisisduetotheextremelylargenumberofdegrees‐of‐freedominneuralnet‐

works,leavingtheirgeneralisationabilityvulnerabletoeffectssuchasoverfitting.

However,thereremainmanydomainswhereextensivedatagatheringisnotalwaysap‐

propriate,affordable,orevenfeasible.Furthermore,dataisgenerallyorganisedinmore

complicatedkindsofstructure—whichmostexistingapproacheswouldsimplydiscard.

Examplesofsuchtasksareabundantinthebiomedicalspace;withe.g.smallnumbersof

subjectsavailableforanygivenclinicalstudy,orrelationshipsbetweenproteinsspecified

viainteractionnetworks.Ihypothesisethat,ifdeeplearningistoreachitsfullpotential

insuchenvironments,weneedtoreconsider“hard‐coded”approaches—integratingas‐

sumptionsaboutinherentstructureintheinputdatadirectlyintoourarchitecturesand

learningalgorithms,throughstructuralinductivebiases.Inthisdissertation,Idirectlyval‐

idate