RFI Identification and Automatic Flagging - Science Website

TFCrop: sliding-win stat + exte

cmdlist = [ “ spw='4' mode='tfcrop' usewindows
“ spw='4' mode='extend' growtime

flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

end, H-S

stats='sum' extendflags=F “,
e=30.0 extendpols=T “ ]

t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 26

Rflag: defaults, H-S

cmdlist = [ “ spw='4' mode='rflag' extendflags=F
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

F“ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 27

Rflag: lower threshold, H-S

cmdlist = [ “ spw='4' mode='rflag' freqdevscale
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

e=3.0 extendflags=F“ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 28

Rflag: lower threshold + exte

cmdlist = [ “ spw='4' mode='rflag' freqdevscale
“ spw='4' mode='extend' growtime

flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

end, H-S

e=3.0 extendflags=F “,
e=30.0 extendpols=T “ ]

t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 29

Rflag: lower threshold + exte

cmdlist = [ “ spw='4' mode='rflag' freqdevscale
“ spw='4' mode='extend' growtime

flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

end + extend, H-S

e=3.0 extendflags=F “,
e=30.0 extendpols=T flagnearfreq=T “ ]

t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 30

TFCrop (narrow spiky RFI): d

cmdlist = [ “ spw='9' mode='tfcrop' extendflags=
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

defaults, no H-S

=F“ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 31

TFCrop (narrow spiky RFI): e

cmdlist = [ “ spw='9' mode='tfcrop' extendflags=
“ spw='9' mode='extend' growtime

flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

extend, no H-S

=F “,
e=50.0 extendpols=T ]

t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 32

TFCrop (narrow spiky RFI): sli
H-S

cmdlist = [ “ spw='9' mode='tfcrop' usewindow
“ spw='9' mode='extend' growtime

flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

iding-win stat + extend, no

wstats='sum' extendflags=F “ ,
e=50.0 extendpols=T “ ]

t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 33

Rflag (narrow spiky RFI): def

cmdlist = [ “ spw='9' mode='rflag' extendflags=F
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

faults, no H-S

F“]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 34

Rflag (narrow spiky RFI): low

cmdlist = [ “ spw='9' mode='rflag' freqdevscale
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

wer threshold, no H-S

e=4.0 extendflags=F “ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 35

Rflag (narrow spiky RFI): low
no H-S

cmdlist = [ “ spw='9' mode='rflag' freqdevscale
“ spw='9' mode='extend' growtime

flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

wer threshold + extend,

e=4.0 extendflags=F “ ,
e=50.0 flagneartime=T extendpols=T ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 36

TFCrop (narrow spiky RFI): d

cmdlist = [ “ spw='9' mode='tfcrop' extendflags=
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

defaults, H-S

=F“ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 37

TFCrop (narrow spiky RFI): e

cmdlist = [ “ spw='9' mode='tfcrop' extendflags=
“ spw='9' mode='extend' growtime

flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

extend, H-S

=F “ ,
e=50.0 extendpols=T flagneartime=T“ ]

t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 38

Rflag (narrow spiky RFI): def

cmdlist = [ “ spw='9' mode='rflag' extendflags=F
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

faults, H-S

F“]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 39

Rflag (narrow spiky RFI): exte

cmdlist = [ “ spw='9' mode='rflag' extendflags=F
“ spw='9' mode='extend' growtime

flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

end, H-S

F“,
e=50.0 extendpols=T flagneartime=T“ ]

t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 40

TFCrop (Broadband Intermitt

cmdlist = [ “ spw='5' mode='tfcrop' extendflags=
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

tent RFI): defaults, no H-S

=F“ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 41

TFCrop (B.I. RFI): reduced po
thresholds + poly-fit in time,

cmdlist = [ “ spw='5' mode='tfcrop' maxnpieces
timefit='poly' extendflags=F “ ]
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

oly-fit pieces + lower
no H-S

s=4 timecutoff=2.5 freqcutoff=3.0

t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 42

TFCrop (B. I. RFI): reduced p
thresholds + poly-fit in time +

cmdlist = [ “ spw='5' mode='tfcrop' maxnpieces
timefit='poly' extendflags=F“, “ spw='5' mode
growfreq=50.0 “ ]
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

poly-fit pieces + lower
+ extend, no H-S

s=4 timecutoff=2.5 freqcutoff=3.0
e='extend' growtime=50.0 extendpols=T

t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 43

TFCrop (B. I. RFI): reduced p
thresholds + poly-fit in time +

cmdlist = [ “ spw='5' mode='tfcrop' maxnpieces
timefit='poly' extendflags=F“, “ spw='5' mode
growfreq=50.0 flagnearfreq=T flagneartime=
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

poly-fit pieces + lower
+ extend + extend, no H-S

s=4 timecutoff=2.5 freqcutoff=3.0
e='extend' growtime=50.0 extendpols=T

=T “ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 44

Rflag (Broadband Intermitten

cmdlist = [ “ spw='5' mode='rflag' extendflag=F “
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

nt RFI): defaults, no H-S

“]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 45

Rflag (B.I. RFI): lower thresho

cmdlist = [ “ spw='5' mode='rflag' freqdevscale
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

olds, no H-S

e=3.0 timedevscale=3.0 extendflag=F “ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 46

Rflag (B.I. RFI): lower thresho

cmdlist = [ “ spw='5' mode='rflag' freqdevscale
“ spw='5' mode='extend' growtime

flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

olds + extend, no H-S

e=3.0 timedevscale=3.0 extendflags=F“ ,
e=50.0 growfreq=30.0 extendpols=T “ ]

t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 47

Choosing what to do.... and

Strong spiky RFI TFCrop ( search for sp
smooth base, per base

Good

Noisy RFI Good if spikes are brigh
Not good for low noisy
Broadband RFI
Un-calibrated data Not robust, but possible
Extended emission of polynomial fit (maxnp

Yes, can fit underlying b

No problem, since each
treated separately.

One suggested usage : TFCrop on uncalibra
(to catch brighte

Hanning Smoothing : Helps for strong RFI wh
But, widens RFI and ca

VLA Data Reduction Workshop -

d when...

pikes above Rflag ( use local stats vs global
eline ) stats to find outliers )

Good, but continuous RFI (time
or freq) needs tuning.

ht enough. Good
RFI

e with tuning Good if RFI looks noisy.
Continuous RFI needs tuning.
pieces)

bandshape No, it needs a flat base

h baseline is Stats are biased by high flux on
short baselines (use uvrange, or
operate on residual data)

ated data + Rflag on calibrated target data

est RFI) (to catch rest of RFI)

here ripples are seen in surrounding channels.
an cause overflagging if there is an ‘RFI forest’.

- 27-31 October 2014 - Socorro 48

Summary

– Automatic Flagging options exist.
– They all need tuning. Usually, one s

=> Look at small pieces of your da
=> Use plotms or viewer or flagda

and try different flagging setup
=> Defaults will not suffice for all c

parameters.

– Use batch-modes in flagdata/flagcm
– Documentation : CASA – Using Cas

AIPS – rflag

VLA Data Reduction Workshop -

setup per SPW or band
ata, and decide flagging strategy
ata(action='calculate', display='data')
ps.
cases, experiment with various

md when relevant.
sa – Other Documentation – Flagging

- 27-31 October 2014 - Socorro 49