improves

BiomedicalSignalProcessingandControl8(2013)400–408

ContentslistsavailableatSciVerScienceDirect

BiomedicalSignalProcessingandControl

journalhomepage:/locate/bspc

Multimodalinformationimprovestherapiddetectionofmentalfatigue

Franc¸oisLaurenta,MarioValderramab,MichelBesrvea,MathiasGuillardc,Jean-PhilippeLachauxd,

JacquesMartineriea,GenevièveFlorencec,∗

aEquipeCogimage(ex-LENA,UPR640),CRICMUMR7225/UMR-S975,UPMC/CNRS/INSERM,47boulevarddel’Hôpital,75651ParisCedex13,France

bUniversityofLosAndes,DepartmentofBiomedicalEngineering,Cra1N

◦

18A-12,Bogotá,Colombia

cInstitutdeRechercheBiomédicaledesArmées(IRBA),DépartementENOP,BP73,91223Brétigny-sur-OrgeCedex,France

dINSERMU821,CentrehospitalierLeVinatier,95boulevardPinel,69675BronCedex,France

articleinfo

Articlehistory:

Received20January2012

Receivedinrevidform8November2012

Accepted22January2013

Availableonline7March2013

Keywords:

EEG

ECG

EOG

Classification

Mentalfatigue

Taskswitching

abstract

Oneofthemajorchallengeinthedetectionofmentalstatesisimprovingtheaccuracyofbrainactivity-

sdthesuitability,for

real-timementalfatiguedetection,ofEEG,EOGandECGmeasurements,takenparatelyortogether.

hparticipant,theblockwiththelowest

errorratefromthefirsttwoblocksandtheblockwiththehighesterrorratefromthelastthreeblocks

werediscriminatedwithamachinelearningalgorithm(supportvectormachine).Theclassificationscores

obtainedwithECGorEOGweregreaterthanwouldbeexpectedbychance(>50%)fortimewindowsof

thebestsinglemodeofdetection,withclassificationscoresrangingfrom80±3%

witha4stimewindowto94±2%itionofECGandEOGfeaturestoEEG

featuressignificantlyincreadclassificationscoresforshorttimewindows(e.g.,to86±3%witha4stime

window,p<0.001).Forshorttimewindows(upto12s),ECGsignificantlyincreadthediscriminatory

powerofEEG,esultsdemonstratethatmentalstatedetectiononthebasisof

extracerebralmeasurementsisfeasibleandthatacombinationofEEGandECGisparticularlyappropriate

fortherapiddetectionofmentalfatigue.

©htsrerved.

uction

Mentalfatiguehasbeendefinedasastateresultingfromthe

prolongedactivityofthebrain,characterizedbybothdecliningper-

formanceandasubjectivefeeling[1].Fatigueisamajorhuman

factorinthesafetyoftransportationsystems[2],andmanystudies

aclas-

sifificationtools

canbeudtoevaluatethepredictivepoweroffunctionalsignals

andarerelevantbecauoftheirpotentialforapplicationtothe

real-timepredictionofbrainstates[3].

Manystudieshaveshownthatelectroencephalographic(EEG)

featuresarerelevantforthedetectionofmentalfatigue[4–7].How-

ever,EEGsaretime-consumingtocarryoutandEEGrecordings

areaffectedbyenvironmentalelectromagneticfields(suchalec-

triclinenoi,noifromelectronicequipment,andvideodisplay

∗

Correspondingauthor.

E-mailaddress:t@(t),

mvalderr@(rama),ve@

(ve),rd@(rd),x@

(J.-x),erie@(erie),

genevieve.florence@(ce).

monitors).ItisthereforenotalwayspossibletocarryoutEEGmea-

sledtoasssmentsof

theufulnessofelectricextracerebralmeasurements,suchalec-

trooculography(EOG)andelectrocardiography(ECG),fordetecting

easurementscanbecarriedoutmoreeasily

r,onlyafewstud-

ieshaveshownthetechniquestobensitiveforthedetection

ofmentalfatigue:anincreainblinkrateasafunctionoftime-

on-taskhasbeenreported[8]andblinkamplitudehasalsobeen

showntobeagoodpredictorofanincreainerrorratedueto

acombinationofsleepdeprivationandtime-on-taskinpilots[9].

Indrivers,asignificantlinearrelationshiphasbeenfoundbetween

thedistancedrivenandthepowerspectrumofheartratevariability

(HRV)analysisinthe0.05–0.15Hzband[10].

EEGandEOGmeasurementscanprovideindicatorsofsleepiness

[11,12],whichis,however,amentalstatedifferentfrommen-

talfatigue,inwhichthereisnotnecessarilyapropensitytofall

asleep[13].Powerspectrumcomponentsofheartratevariability,

theP300componentandwaveletparametersofEEGhavealready

beenstudied,inanalysoftheimpactofprolongedvisualdisplay

r,thevariableshavebeenmeasured

independently,andonlybeforeandafterthefatigue-inducingtask

[14].

1746-8094/$–efrontmatter©htsrerved.

/10.1016/.2013.01.007

tetal./BiomedicalSignalProcessingandControl8(2013)400–408401

EEG,EOGandECGmeasurementshavebeenassdtogether,

todeterminetheaccuracyofaclassifierfordetectingdifferentlev-

elsofmentalworkload[15],butnottoevaluatementalfatigue.

Althoughtheauthorsofthispreviousstudyudtheexpression

“mentalworkload”inthetitleoftheirpaper,theydidnotclearly

defid,theyudtheexpression“functional

stateofthehumanoperator”.Theymodifiedthis“functionalstate”

byincreasingthedifficultyoftheNASAMulti-AttributeTaskBattery

(MATB)applied,assumingthatincreasingthedifficultyofthetask

wouldalter“functionalstate”(e.g.,increathementalworkload).

Theydidnotreallystudymentalfatigue,becautheparticipants

performedtheMATBfor15minonly,inthree5-min-longtests

(balineconditions,lowdifficultyandhighdifficultylevel).

Anotherstudydemonstratedthepotentialofacombinationof

scalpandintracranialEEGandECGfeaturesforidentifyingindi-

vidualsathighriskofepilepticizures,whichcanbeviewedas

anothertypeofmentalstate[16].ThecombineduofEEG,EOG

andECGforthedetectionofmentalfatiguehasneverbeeninves-

tigated,toourknowledge.

Weaddresdfourissues:(1)CaneachoftheEOG,ECGandEEG

methods,udindividually,detectmentalfatiguethroughtheir

respectivemeasurements?Inotherwords,dotheyperformbetter

thanwouldbeexpectedbychance?(2)Whatadvantageistherein

usingallthreemodesofmeasurementtogether?Doesthesimul-

taneousuofEOG,ECGandEEGimprovethedetectionofmental

fatigueoverthatachievedwithEEGalone?(3)Ifso,isthisimprove-

mentduetothecomplementarynatureoftheinformationprovided

bythedifferenttypesofrecording,ashoped,ordoesitresultfrom

moremechanisticeffectsoftheclassificationmethodology,such

asdifferencesinthenumberofquantificationvariables?(4)What

contributionstoEOGandECGfeaturesmaketothemultimodal

detector?Shouldboththeextracerebralmodesofasssmentbe

consideredinadditiontoEEGorcanasimilarpredictionperfor-

mancebeachievedwithjusttwomodesofasssment?

Forthefirsttwooftheissues,wealsostudiedtheinfluence

ofwindowduration(4–30s),whichmaybedeterminantforreal-

,itappearsreasonabletosuggestthatshorter

windowdurationsarelikelytobeassociatedwithearlierdetection.

Finally,foramorephysiologicalapproachtomentalfatigue,we

identifiedthefeatureswiththehighestdiscriminatingpower,for

awindowof30s.

Thisstudywasthusdesignedtodeterminewhetherthecombi-

nationofdifferenttypesofrecordingcouldimprovethedetection

ot

ourobjectivetoprovideadetailedanalysisoftheeffectofmental

fatigueineachofthedifferenttypesofrecording.

Mentalfatiguewasinducedbyincreasingtime-on-task,witha

previouslyvalidatedparadigm[17].

s

mentalparadigm

Thirteenright-handedmalesubjectstookpartinatask-

switchingexperimentcarriedoutinaccordancewiththeHelsinki

asrecordedwitha32-electrodeActiCapTM

andaBrainAmpTMsystem,averticalEOGwithrecordedwithtwo

electrodesaroundthedominanteyeandanECGwasrecordedwith

aleadbetweenabottom-leftcostallocationandatop-leftlocation

intheback.

Subjectswerefirstsubjectedtoatrainingssion,inwhichthey

carriedoutablockoftask(660stimuli)withauditoryfeedbackif

enperformedahalf-blockoftask(330

stimuli)ainingssionwascarriedoutin

tsthenhadanadditionaltrainingssion(one

–letterpairswereprentedinfourquad-

rants,tswereinstructedtorespondpressabutton

indicatingwhetherthenumberwasoddorevenorwhethertheletterwasaconso-

nantoravowel,dependingonthelocationofthenumber–letterpair.

blockoftaskswithfeedbackincasoferror)beforethession.

Theywereaskedtorespondascorrectlyandasquicklyaspossible

toavisualstimulusconsistingofanumberandaletter(Fig.1).Each

stimuluswasdisplayedinoneofthefourquadrantsofasquarebox

placedwithinthesubject’scentralfijectsud

twofimuliappearingin

theuppertwoquadrants,thesubjectswereaskedtorespondtothe

numberstheysaw(rightbuttonpressforevennumbers,leftbutton

pressforoddnumbers).Forstimuliappearinginthelowerquad-

rants,subjectswereaskedtorespondtotheletterstheysaw(right

buttonpressforaconsonant,leftbuttonpressforavowel).Once

thesubjecthadrespondedor2500mshadelapdsincethestim-

uluswasfirstdisplayed,anewstimulusappearedintheadjacent

quadrant,pon-stimulusinterval

(RSI)wasvariable(300,600or1500ms).Eachssionconsisted

ofsixconcutiveblocksof660trialach(approximately20min)

andallssionstookplaceintheafternoon(from2pmto5pm).

Beforethestartofthessionandaftereachblock,thesubjects

filledinthePearson–Byarsfatiguechecklist[18].

mentalstates

Badoftheresultsofpreviouxperiments,wedefinedthe

twomentalstatesofinterestascorrespondingtotwodifferent

blocksoftask[6,17,19].Weconsiderederrorratetobetheprincipal

tivefatiguescoresincreadduring

theexperiment,buttheerrorratedidnotincreamonotonically.

Accordingly,bycontrasttopreviouxperiments,wedidnotcon-

siderthefirstblockasthe“no-fatigueblock”andthelastblock

asthe“fatigueblock”.Instead,weudanapproachinwhichthe

blocksconsidereddependedontheperformanceoftheparticipant.

Foreachsubject,wedefinedtheno-fatigueblockastheblockfrom

thefirsttwoforwhichperformancewashighest,andwedefinedthe

fatigueblockastheblockfromthelastthreeforwhichperformance

waslowest.

equantification

Aslidingtime-windowwasudtoextractelectrophysiological

veraltime-windowdurations,

ainedinthediscussion,30sshould

beasuitableperiodforthecorrectestimationofECGspectrum

features,whereasEEGrhythmsareusuallyobrvedwithsmaller

edtoensurethatatleast10cycleswere

hereforenecessarytou

timewindowsofatleast3s,takingthelowestfrequencystudied

sivewindowsoverlappedby50%.

tures

WecorrectedtheEEGforeyeblinks,gmentingthesignalevery

2sandusingaPCA(principalcomponentanalysis)-badmethod.

tetal./BiomedicalSignalProcessingandControl8(2013)400–408

Wediscardedprincipalcomponentsfeaturingacorrelationwith

anyEOGchannelgreaterthan0.7[20],andctionswithhighlev-

ulatedacommonaveragereference

byexcludingthemostperipheralelectrodes(Fp1,Fp2,F7,F8,T7,T8,

TP9,TP10,P7,andP8)andweappliedthisreferencetothemeasure-

mentsofalltheelectrodes,toeliminatebackgroundnoiwithout

disminatingelectricalactivitiesfrompericranialmuscles.

Meanamplitudesinthe3–7Hz(Â),7–13Hz(˛),and13–18Hz

(ˇ)frequencybandswerecalculatedforeachEEGelectrode(96

features=32electrodes×3features),fromtheamplitudeofeach

signal/electrode:

MA=

1

T