irc2 speed test (fGpu={0,1} x fParfor={0,1})

# Environment
– XEON 20 cores 3.4GB
– 256GB ram
– SSD

# GPU ON, parfor ON
Recording format
Recording file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw.mda
Probe file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\geom.csv
Recording Duration: 1200.0s
Data Type: int16
#Channels in file: 64
#Sites: 64
#Shanks: 1
Pre-processing
Filter type: bandpass
Filter range (Hz): 300.0-6000.0
Common ref: mean
FFT threshold: 8
Events
#Spikes: 497473
Feature extracted: gpca
#Sites/event: 14
maxDist_site_um: 50
maxDist_site_spk_um: 75
#Features/event: 20
Cluster
#Clusters: 96
#Unique events: 493550
min. spk/clu: 30
Cluster method: drift-knn
knn: 30
nTime_clu: 4
nTime_drift: 60
fSpatialMask_clu: 0
Auto-merge
delta_cut: 1.000
maxWavCor: 0.990
Runtime (s)
Detect + feature (s): 49.9s
Cluster runtime (s): 9.3s
merge runtime (s): 12.4s
Total runtime (s): 71.6s
Runtime speed: x16.8 realtime
memory usage (GiB): 4.716
detect(GiB): 4.716
sort(GiB): 0.742
Execution
fGpu (GPU use): 1
fParfor (parfor use): 1
Parameter file: C:\tmp\irc2\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw_geom.prm

# GPU ON, parfor OFF
Recording format
Recording file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw.mda
Probe file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\geom.csv
Recording Duration: 1200.0s
Data Type: int16
#Channels in file: 64
#Sites: 64
#Shanks: 1
Pre-processing
Filter type: bandpass
Filter range (Hz): 300.0-6000.0
Common ref: mean
FFT threshold: 8
Events
#Spikes: 497473
Feature extracted: gpca
#Sites/event: 14
maxDist_site_um: 50
maxDist_site_spk_um: 75
#Features/event: 20
Cluster
#Clusters: 96
#Unique events: 493550
min. spk/clu: 30
Cluster method: drift-knn
knn: 30
nTime_clu: 4
nTime_drift: 60
fSpatialMask_clu: 0
Auto-merge
delta_cut: 1.000
maxWavCor: 0.990
Runtime (s)
Detect + feature (s): 116.5s
Cluster runtime (s): 19.6s
merge runtime (s): 12.4s
Total runtime (s): 148.5s
Runtime speed: x8.1 realtime
memory usage (GiB): 0.989
detect(GiB): 0.989
sort(GiB): 0.618
Execution
fGpu (GPU use): 1
fParfor (parfor use): 0
Parameter file: C:\tmp\irc2\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw_geom.prm

# GPU OFF, parfor ON
Recording format
Recording file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw.mda
Probe file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\geom.csv
Recording Duration: 1200.0s
Data Type: int16
#Channels in file: 64
#Sites: 64
#Shanks: 1
Pre-processing
Filter type: bandpass
Filter range (Hz): 300.0-6000.0
Common ref: mean
FFT threshold: 8
Events
#Spikes: 497473
Feature extracted: gpca
#Sites/event: 14
maxDist_site_um: 50
maxDist_site_spk_um: 75
#Features/event: 20
Cluster
#Clusters: 96
#Unique events: 493638
min. spk/clu: 30
Cluster method: drift-knn
knn: 30
nTime_clu: 4
nTime_drift: 60
fSpatialMask_clu: 0
Auto-merge
delta_cut: 1.000
maxWavCor: 0.990
Runtime (s)
Detect + feature (s): 71.4s
Cluster runtime (s): 11.0s
merge runtime (s): 12.1s
Total runtime (s): 94.6s
Runtime speed: x12.7 realtime
memory usage (GiB): 4.720
detect(GiB): 4.720
sort(GiB): 0.700
Execution
fGpu (GPU use): 0
fParfor (parfor use): 1
Parameter file: C:\tmp\irc2\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw_geom.prm

# GPU OFF, parfor OFF
Recording format
Recording file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw.mda
Probe file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\geom.csv
Recording Duration: 1200.0s
Data Type: int16
#Channels in file: 64
#Sites: 64
#Shanks: 1
Pre-processing
Filter type: bandpass
Filter range (Hz): 300.0-6000.0
Common ref: mean
FFT threshold: 8
Events
#Spikes: 497473
Feature extracted: gpca
#Sites/event: 14
maxDist_site_um: 50
maxDist_site_spk_um: 75
#Features/event: 20
Cluster
#Clusters: 96
#Unique events: 493638
min. spk/clu: 30
Cluster method: drift-knn
knn: 30
nTime_clu: 4
nTime_drift: 60
fSpatialMask_clu: 0
Auto-merge
delta_cut: 1.000
maxWavCor: 0.990
Runtime (s)
Detect + feature (s): 236.1s
Cluster runtime (s): 48.3s
merge runtime (s): 12.0s
Total runtime (s): 296.4s
Runtime speed: x4.0 realtime
memory usage (GiB): 1.013
detect(GiB): 1.013
sort(GiB): 0.614
Execution
fGpu (GPU use): 0
fParfor (parfor use): 0
Parameter file: C:\tmp\irc2\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw_geom.prm

Flatiron weekly progress: Sep30-Oct4

# ironclust v2
– […] run memory benchmark
– […] update spikeforest website benchmark
– [x] plot quality comparison
– [x] update the spikeforest wrapper

# Dan English
– create SNR distribution plot
– compare with others dataset

# paper writing
– jeremy flow chart
– contribute to spikeforest

# misc
– [x] ottawa travel reimbursement

# Computer maintenance
## Ubuntu
[x] VNC viewer installation
– [x] yakuake terminal sudo apt-get install yakuake
– [x] vscode (got stuck, can’t install code on terminal)

## Windows
– [x] Windows 10 install on moneyboxwin
– [x] office 365, TreeSizeFree, KarenReplicator
– [x] Copy 5GB backup drive

## Disk drive
– […] Initialize RAID48GB
– [ ] Copy recordings to 48GB
– [ ] Copy personal files to 48GB
– [ ] Build 60 GB Linux partition, put in recordings
– [x] Build 48 GB backup drive (RAID5), put in all recordings

irc2 development log

auto-merge: using feature RMS instead of waveform correlation

dataset: hybrid_janelia_static
(64ch, 1200s, 72 units, 30KS/s)
fGpu=0fGpu=1
fParfor=0Runtime (s):
Detect + feature (s): 132.0s
Cluster (s): 94.0s
Automerge (s): 17.4s
Total runtime (s): 243.4s
Runtime speed x4.9 realtime
memory usage (GiB):
detect(GiB): 0.900
sort(GiB): 0.380
Runtime (s):
Detect + feature (s): 57.5s
Cluster (s): 30.6s
Automerge (s): 19.6s
Total runtime (s): 107.6s
Runtime speed x11.2 realtime
memory usage (GiB):
detect(GiB): 1.090
sort(GiB): 0.482
fParfor=1
(4 local workers)
Runtime (s):
Detect + feature (s): 86.9s
Cluster (s): 48.7s
Automerge (s): 14.1s
Total runtime (s): 149.7s
Runtime speed x8.0 realtime
memory usage (GiB):
detect(GiB): 4.192
sort(GiB): 0.577
CRASHED
fParfor=1
(20 local workers)
Runtime (s):
Detect + feature (s): 76.1s
Cluster (s): 22.1s
Automerge (s): 10.7s
Total runtime (s): 108.9s
Runtime speed x11.0 realtime
memory usage (GiB):
detect(GiB): 4.169
sort(GiB): 0.560
CRASHED
fParfor=1
(20 remote workers)
Runtime (s):
Detect + feature (s): 58.4s
Cluster (s): 19.2s
Automerge (s): 9.3s
Total runtime (s): 86.9s
Runtime speed x13.8 realtime
memory usage (GiB):
detect(GiB): 4.221
sort(GiB): 0.743
**-p gpu=”gpures:2″**
Runtime (s):
Detect + feature (s): 38.2s
Cluster (s): 12.4s
Automerge (s): 9.9s
Total runtime (s): 60.5s
Runtime speed x19.8 realtime
memory usage (GiB):
detect(GiB): 4.174
sort(GiB): 0.334

irc2 post merging using position and amplitude of clusters

use gaussian kernel smoothing (make sure i get half a fall off at half the mindist). normalize by projecting a uniform field and ensure uniform field back.

advantage of this approach is robustness to where the peak site is located in determining the peak location.

gaussian kernel convolved, maximum slope at the minimum separation distance (sigma=d_min)
inferring spike position using PC1 is more precise than using other components
Great study music helping me to focus

irc2 development log

# fixed automerging issue
– Spike indexing was incorrect when extracting trPc 3D array.
– waveform shifting produced comparable result

# todo
– [x] compute rho and delta using parallel resources
– [x] compare performance between irc and irc2
– [ ]add drift correction and compare drift performance

# Runtime comparison
– dataset: static_siprobe\rec_64c_1200s_11
– irc.m: 123s, mean accuracy: 89.8, 62 above .8 accuracy, 1.8GB
– irc2.m (fGpu=1,fParfor=0): 64s, mean accuracy: 89.3, 62 above .8 accuracy, .776GB

# irc2.m speed test (drift correction not implemented yet)
-fGpu=1, fParfor=0: 64s
-fGpu=0, fParfor=0: 389s
-fGpu=0, fParfor=1: 158.6s (20 nodes, local)

# irc2.m test on linux workstation
-fGpu=0, fParfor=0: 292.5s
-fGpu=0, fParfor=1: 133s (20 nodes, remote)
-fGpu=1, fParfor=0: 45s

Losing weight by walking to work and back

2019 Sep 22: 93 KG

action: walked for an hour to get to work. Will do the same on the way back. That’s two hours of walking per day. I will also save $20 a day not taking ferry. I will lose 1KG, save $20, and spend extra 1 hour a day commuting to work. and back. In a month I will be 30 KG lighter and $600 richer.