RFI Identification and Automatic Flagging - Science Website

RFI Identification and Autom

Urvashi Rau & Em1
NRA

VLA Data Reduc
27 – 31 Oct

VLA Data Reduction Workshop -

matic Flagging

m1m38a8nMuHezl Momjian 1756 MHz
AO
ction Workshop
tober 2014

- 27-31 October 2014 - Socorro 1

Outline

● RFI at the VLA + Onli
● Automatic Flagging Op
● Some examples of wh1a

VLA Data Reduction Workshop -

ine Flags
ptions and Strategies
1a3t88wMoHrzks when.... 1756 MHz

( 31 slides )

- 27-31 October 2014 - Socorro 2

RFI at the VLA + Online fla

Radio Frequency Interference : Terrestrial,

At L-Band, can use ~500 MHz with very rou
RFI is less at higher frequencies. At X-ban

Online + Deterministic Flags :

– slew, subreflector error, focus error
– operator logs of known bad antenna
– shadowing between antennas (elev
– exact zeros (from the correlator)
– known frequency ranges with bad R

VLA Data Reduction Workshop -

ags

Satellites, etc..
ugh flagging, ~800 MHz if done carefully.
nd and beyond, RFI is minimal

https://science.nrao.edu/facilities/vla/ -- Guide to VLA
Observing -- Radio Frequency Interference

as and time-ranges 3
vation-dependent)

RFI

- 27-31 October 2014 - Socorro

Deterministic Flagging Tasks - flagdata +

mode='manual' or mode='unflag' # Use MS-selecti
mode = 'quack' # Flag data at the beginning or e
mode = 'elevation' # Flag data between specified e
mode='shadow' # Flag baselines with shadowed

Autoflag mode = 'clip' # Threshold-based flagging on
mode='tfcrop' # Automatic flagging : Find outl
mode='rflag' # Find outliers based on sliding
mode='extend' # Grow/extend flags around exi

Operational mode='summary' # Count existing flags and retur

mode = 'list' # Supply a list of flag command

Run-modes : “ apply / calculate “ + runtime “displa

VLA Data Reduction Workshop -

flagcmd

ion syntax to pick subsets to flag/unflag
end of a scan (operate on selected data).
elevation limits (operate on selected data)
d-antennas

data-expressions (ABS_RR, ABS_I, etc)
liers on the 2D time-frequency plane
g-window RMS filters
isting ones.

rn a python-dictionary of counts per antenna, spw, etc
ds, built out of parameters of any other mode.
ay” of data before and after flagging, and reports.

- 27-31 October 2014 - Socorro 4

Managing the Flags - Flagma

Flagging tasks (flagdata, flagcmd), applycal, flagma
These are the state of the flag of the data befo

Flag backups are stored in the directory: mydata.ms
Flag backups are named as ‘flags.the task creating i

flags.flagcmd_1
flags.flagdata_1
flags.before_applycal_1
flags.flagcmd_2
flags.flagdata_2
flags.flagdata_3
flags.mybackup (flagmanager backups get n

Flagmanager is the task to operate on flag backups
mode= 'list' to list existing flagtables
= 'save' to save flag column from vis to
= 'restore' to place the specified flag file
= 'delete' to delete specified flag file
= 'rename' to rename a specified flag file

To restore flags: restores flags.fla
versionname=’flagcmd_1’

VLA Data Reduction Workshop -

anager

anager can make flag backup versions
ore the new flags are applied on the data.
s.flagversions
it_version#’

named by you, e.g. mybackup) 5

o a specified flag file
into vis
e
agcmd_1

- 27-31 October 2014 - Socorro

Flagging Modes : tfcrop

Detect outliers on the 2D time-freq plane.

For each baseline,
(1) Average visibility amplitudes along time dimension to

form an average spectrum
(2) Calculate a robust piece-wise polynomial fit for the

band-shape at the base of RFI spikes
– Calculate 'sigma' of (data - fit).

(3) Flag points deviating from the fit by more than N-sigma
(4) Repeat (1-3) along the other dimension.

- Calculates and applies flags in one pass through the
- Can operate on un-calibrated data and bandshapes
- Flag extension can be done via mode='extend'

VLA Data Reduction Workshop -

MS 6

- 27-31 October 2014 - Socorro

Flagging Modes : tfcrop

ntime = 'scan' or '2.0min' or 120.0 # Time range

combinescans = True/False # if ntime > s

datacolumn = 'data', 'corrected', 'model', 'residual'

correlation = 'ABS_ALL' # Correlation e
Expressions
Functions : 'A
Ex : 'ABS_RR

timecutoff, freqcutoff = 3.0 # Scale-factor fo

timefit, freqfit = 'line' or 'poly' # Fit a straight lin

maxnpieces = 7 # maximum num

flagdimension = 'freqtime' # Directions in w

usewindowstats , halfwin # Options to use

VLA Data Reduction Workshop -

e to use for each chunk
scan, join multiple scans

expression. Augmented MS-selection syntax.
: 'ALL', 'RR','LL', 'I', 'Q','U','V', 'WVR'
ABS', 'ARG', 'RE', 'IM', 'NORM'
R,LL,V'
or deviation from fit, to compute thresholds
ne, or piecewise polynomial
mber of pieces in the polynomial fit (only for 'poly')

which to calculate stats 'timefreq','time','freq'

e sliding-window statistics 'std' and 'sum'

- 27-31 October 2014 - Socorro 7

Flagging Modes : rflag ( A c

Detect outliers based on sliding-window RMS filte

(1) Time-Analysis (for each channel)

(1a) Calculate local RMS of real and imag parts of visibilities w
a sliding time window.

(1b) Calculate the median RMS across time windows, deviatio
local RMS from this median, and the median deviation

(1c) Flag if local RMS > N x (medianRMS + medianDev)

(2) Spectral-Analysis (for each timestep)

(2a) Calculate Avg of real and imag parts of visibilities and th
RMS across channels.

(2b) Calculate the deviation of each channel from this Avg, an
the median-deviation.

(2d) Flag if deviation > N x medianDev

- Can be run with 1 or 2 passes through the data
- Requires calibrated data.
- Flag extension can be done via mode='extend'
- Can flag short baselines if there is extended emission
- Good for low level noisy RFI

VLA Data Reduction Workshop -

close copy of RFLAG from AIPS (E.Greisen, 2011) )

ers.

within
ons of

heir
nd

n 8

- 27-31 October 2014 - Socorro

Flagging Modes : rflag

ntime = 'scan' or '2.0min' or 120.0 # Time range

combinescans = True/False # if ntime > s

datacolumn = 'data', 'corrected', 'model', 'residual'

correlation = 'ABS_ALL' # Correlation e
Expressions
Functions : 'A
Ex : 'ABS_RR

winsize = 3 # size of sliding t

timedev = [ [ 0, 2, 0.0534 ] ] # time-series nois
freqdev = [ ] # spectral noise e

timedevscale = 5 # threshold scalin
freqdevscale = 5 # threshold scaling

spectralmin = 1e+6 # flag whole spectr
spectralmax = 0.0 # flag whole spectr

outfile = 'thresholds.txt' # text file to hold

VLA Data Reduction Workshop -

e to use for each chunk
scan, join multiple scans

expression. Augmented MS-selection syntax.
: 'ALL', 'RR','LL', 'I', 'Q','U','V', 'WVR'
ABS', 'ARG', 'RE', 'IM', 'NORM'
R,LL,V'
time window
se estimate [field, spw, noise-estimate]
estimate
ng for timedev
g for freqdev
rum if freqdev > spectralmin
rum if freqdev < spectralmax
output thresholds from rflag.

- 27-31 October 2014 - Socorro 9

Flagging Modes : extend

Example :
Flag only RR,

+
Growtime = 30.0

+
Extendpols = True

Useful as a
follow-up option
after rflag or tfcrop.

VLA Data Reduction Workshop -

- 27-31 October 2014 - Socorro 10

Flagging Modes : extend

Extend flags along time, frequency, polarizatio

ntime = 'scan' or '2.0min' or 120.0 # Time range

combinescans = True/False # if ntime > s

extendpols = True/False # Extend ac

growtime = 80.0 # For each ch
if more tha

growfreq = 80.0 # For each tim
SPW) if m

growaround = True/False # On the 2D
surroundin

flagneartime = True/False # Flag points
flagnearfreq = True/False # Flag channe

VLA Data Reduction Workshop -

ons (and baselines ). Operates on selected data

e to use for each chunk
scan, join multiple scans
cross all selected correlations
hannel, flag entire timerange (ntime)
an 80% of the data is flagged.
me, flag selected channels (in current
more than 80% of the data is flagged.
time/freq plane, if more than 4
ng points are flagged, flag it.
s before after every flagged point in time
els before/after every flagged one.

- 27-31 October 2014 - Socorro 11

beforeProtecting spectral-lines, d

after spw = ' 9 '

This is relevant to 'clip' , 'exte

VLA Data Reduction Workshop -

during automatic flagging

spw = ' 9 : 0~45; 53~63 '

end' , 'tfcrop' , 'rflag' 12

- 27-31 October 2014 - Socorro

Flagging Modes : list ( fla

Supply a list of flag commands, to be run

- Apply large numbers of online flags (all mod
- Combine modes that read the same data, t
- flagcmd can read flags from XML files and r

Example list of commands ( in a text file ) :

flagdata( vis='xxx.ms', mode='list', inpfile='cmd

mode='manual' antenna='1&2' spw='3' t
mode='manual' antenna='ea05' spw='5:50
mode='manual' antenna='ea01,ea24' spw
mode='quack' quackinterval=5.0
mode='shadow' tolerance=5.0

Example command list ( interactive input ) :

flagdata( vis='xxx.ms', mode='list', inpfile=cmd

cmdlist = [ “mode='manual' antenna='1&2' spw

VLA Data Reduction Workshop -

agdata, flagcmd )

in one single pass through the data.

de='manual') together.
to save on I/O
read/write to the FLAG_CMD MS subtable

ds.txt' )
timerange='12:23:02.3~12.24:45.2'
0~60' scan='2~10'
w='9' correlation='RR,RL'

dlist )
w='3'” , “mode='shadow' tolerance=5.0” ]

- 27-31 October 2014 - Socorro 13

Summary mode - output python

Output : Dictionary of flag counts (vs) antenna,

counts = flagdata(vis='xxx.ms' , mode='summar
print “Counts per spw : “, counts['spw'];
print “Total % flagged : “ , 100.0 * counts['flagge
print “Number of unflagged visibilities in spw 9 :

VLA Data Reduction Workshop -

n dictionary of flag counts

, field, spw, channel, correlation, etc...

ry' , display='report' , spwchan=True );

ed'] / counts['total']
: “, counts['spw']['9']['total'] – counts['spw']['9']['flagged']

- 27-31 October 2014 - Socorro 14

Auto-flagging strategies – a

In general, examine pieces of your data visu
--- one scan --- few baselines --- all s

All current RFI flagging algorithms require tu
– Rflag calculates statistics across all time
Tunable parameters : N-sigma threshold
– TFCrop operates on the time-freq plane
Tunable parameters : N-sigma threshold
– Flag Extension
Tunable parameters : Grow flags in time
extend across co
– AO-Flagger from LOFAR : http://source
Operates on the 2D time-freq plan

VLA Data Reduction Workshop -

all methods need tuning

ually, to decide auto-flagging strategy
spws ---
uning for best results.
es, channels, baselines and pols
ds, sliding-window sizes
e, separately per baseline and correlation.
ds, window sizes, types of time/freq fits.

e and frequency or both,
orrelations, etc....

eforge.net/p/aoflagger/wiki/Home/

ne, based on a robust 2D rubber-sheet fit

- 27-31 October 2014 - Socorro 15

Exam

VLA Data Reduction Workshop -

mples

- 27-31 October 2014 - Socorro 16

TFCrop: defaults, no H-S

flagdata(vis='xxx.ms', spw='4' mode='tfcrop', ac
(or)
cmdlist = [ “ spw='4' mode='tfcrop' extendflags=
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

ction='calculate', display='data', extendflags=F)

=F “ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 17

TFCrop: reduced poly-fit piec

flagdata(vis='xxx.ms', spw='4' mode='tfcrop', ma
display='data')
cmdlist = [ “ spw='4' mode='tfcrop' maxnpieces
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

ces, no H-S

axnpieces=4, action='calculate', extendflags=F,

s=4 extendflags=F“ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 18

TFCrop: reduced poly-fit piece

cmdlist = [ “ spw='4' mode='tfcrop' maxnpieces
extendflags=F “ ]
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

es + sliding-win stat, no H-S

s=4 usewindowstats='std' halfwin=2,

t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 19

TFCrop: reduced poly-fit piece
no H-S

cmdlist = [ “ spw='4' mode='tfcrop' maxnpieces
extendflags=F “, “ spw='4' mode='extend' gro
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

es + sliding-win stat + extend,

s=4 usewindowstats='std' halfwin=2
owtime=60.0 extendpols=T “ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 20

Rflag: defaults, no H-S

cmdlist = [ “ spw='4' mode='rflag' extendflags=F“
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

“]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 21

Rflag: lower threshold, no H-

cmdlist = [ “ spw='4' mode='rflag' freqdevscale=
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

-S

=4.0 extendflags=F“ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 22

Rflag: lower threshold + exte

cmdlist = [ “ spw='4' mode='rflag' freqdevscale
“ spw='4' mode='extend' growtime=

flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist,

VLA Data Reduction Workshop -

end, no H-S

e=4.0 extendflags=F“,
=30.0 extendpols=T ]

, action='calculate', display='data')

- 27-31 October 2014 - Socorro 23

TFcrop: defaults, H-S

cmdlist = [ “ spw='4' mode='tfcrop' extendflags=
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

=F“ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 24

TFCrop: sliding-win stat, H-S

cmdlist = [ “ spw='4' mode='tfcrop' usewindows
flagdata(vis='xxx.ms', mode='list', inpfile=cmdlist

VLA Data Reduction Workshop -

stats='sum' extendflags=F “ ]
t, action='calculate', display='data')

- 27-31 October 2014 - Socorro 25