James Jun

# Environment
– XEON 20 cores 3.4GB
– 256GB ram
– SSD

# GPU ON, parfor ON
Recording format
Recording file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw.mda
Probe file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\geom.csv
Recording Duration: 1200.0s
Data Type: int16
#Channels in file: 64
#Sites: 64
#Shanks: 1
Pre-processing
Filter type: bandpass
Filter range (Hz): 300.0-6000.0
Common ref: mean
FFT threshold: 8
Events
#Spikes: 497473
Feature extracted: gpca
#Sites/event: 14
maxDist_site_um: 50
maxDist_site_spk_um: 75
#Features/event: 20
Cluster
#Clusters: 96
#Unique events: 493550
min. spk/clu: 30
Cluster method: drift-knn
knn: 30
nTime_clu: 4
nTime_drift: 60
fSpatialMask_clu: 0
Auto-merge
delta_cut: 1.000
maxWavCor: 0.990
Runtime (s)
Detect + feature (s): 49.9s
Cluster runtime (s): 9.3s
merge runtime (s): 12.4s
Total runtime (s): 71.6s
Runtime speed: x16.8 realtime
memory usage (GiB): 4.716
detect(GiB): 4.716
sort(GiB): 0.742
Execution
fGpu (GPU use): 1
fParfor (parfor use): 1
Parameter file: C:\tmp\irc2\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw_geom.prm

# GPU ON, parfor OFF
Recording format
Recording file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw.mda
Probe file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\geom.csv
Recording Duration: 1200.0s
Data Type: int16
#Channels in file: 64
#Sites: 64
#Shanks: 1
Pre-processing
Filter type: bandpass
Filter range (Hz): 300.0-6000.0
Common ref: mean
FFT threshold: 8
Events
#Spikes: 497473
Feature extracted: gpca
#Sites/event: 14
maxDist_site_um: 50
maxDist_site_spk_um: 75
#Features/event: 20
Cluster
#Clusters: 96
#Unique events: 493550
min. spk/clu: 30
Cluster method: drift-knn
knn: 30
nTime_clu: 4
nTime_drift: 60
fSpatialMask_clu: 0
Auto-merge
delta_cut: 1.000
maxWavCor: 0.990
Runtime (s)
Detect + feature (s): 116.5s
Cluster runtime (s): 19.6s
merge runtime (s): 12.4s
Total runtime (s): 148.5s
Runtime speed: x8.1 realtime
memory usage (GiB): 0.989
detect(GiB): 0.989
sort(GiB): 0.618
Execution
fGpu (GPU use): 1
fParfor (parfor use): 0
Parameter file: C:\tmp\irc2\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw_geom.prm

# GPU OFF, parfor ON
Recording format
Recording file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw.mda
Probe file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\geom.csv
Recording Duration: 1200.0s
Data Type: int16
#Channels in file: 64
#Sites: 64
#Shanks: 1
Pre-processing
Filter type: bandpass
Filter range (Hz): 300.0-6000.0
Common ref: mean
FFT threshold: 8
Events
#Spikes: 497473
Feature extracted: gpca
#Sites/event: 14
maxDist_site_um: 50
maxDist_site_spk_um: 75
#Features/event: 20
Cluster
#Clusters: 96
#Unique events: 493638
min. spk/clu: 30
Cluster method: drift-knn
knn: 30
nTime_clu: 4
nTime_drift: 60
fSpatialMask_clu: 0
Auto-merge
delta_cut: 1.000
maxWavCor: 0.990
Runtime (s)
Detect + feature (s): 71.4s
Cluster runtime (s): 11.0s
merge runtime (s): 12.1s
Total runtime (s): 94.6s
Runtime speed: x12.7 realtime
memory usage (GiB): 4.720
detect(GiB): 4.720
sort(GiB): 0.700
Execution
fGpu (GPU use): 0
fParfor (parfor use): 1
Parameter file: C:\tmp\irc2\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw_geom.prm

# GPU OFF, parfor OFF
Recording format
Recording file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw.mda
Probe file: C:\tmp\groundtruth\hybrid_synth\drift_siprobe\rec_64c_1200s_11\geom.csv
Recording Duration: 1200.0s
Data Type: int16
#Channels in file: 64
#Sites: 64
#Shanks: 1
Pre-processing
Filter type: bandpass
Filter range (Hz): 300.0-6000.0
Common ref: mean
FFT threshold: 8
Events
#Spikes: 497473
Feature extracted: gpca
#Sites/event: 14
maxDist_site_um: 50
maxDist_site_spk_um: 75
#Features/event: 20
Cluster
#Clusters: 96
#Unique events: 493638
min. spk/clu: 30
Cluster method: drift-knn
knn: 30
nTime_clu: 4
nTime_drift: 60
fSpatialMask_clu: 0
Auto-merge
delta_cut: 1.000
maxWavCor: 0.990
Runtime (s)
Detect + feature (s): 236.1s
Cluster runtime (s): 48.3s
merge runtime (s): 12.0s
Total runtime (s): 296.4s
Runtime speed: x4.0 realtime
memory usage (GiB): 1.013
detect(GiB): 1.013
sort(GiB): 0.614
Execution
fGpu (GPU use): 0
fParfor (parfor use): 0
Parameter file: C:\tmp\irc2\hybrid_synth\drift_siprobe\rec_64c_1200s_11\raw_geom.prm

# ironclust v2
– […] run memory benchmark
– […] update spikeforest website benchmark
– [x] plot quality comparison
– [x] update the spikeforest wrapper

# Dan English
– create SNR distribution plot
– compare with others dataset

# paper writing
– jeremy flow chart
– contribute to spikeforest

# misc
– [x] ottawa travel reimbursement

# Computer maintenance
## Ubuntu
– [x] VNC viewer installation
– [x] yakuake terminal sudo apt-get install yakuake
– [x] vscode (got stuck, can’t install code on terminal)

## Windows
– [x] Windows 10 install on moneyboxwin
– [x] office 365, TreeSizeFree, KarenReplicator
– [x] Copy 5GB backup drive

## Disk drive
– […] Initialize RAID48GB
– [ ] Copy recordings to 48GB
– [ ] Copy personal files to 48GB
– [ ] Build 60 GB Linux partition, put in recordings
– [x] Build 48 GB backup drive (RAID5), put in all recordings

auto-merge: using feature RMS instead of waveform correlation

dataset: hybrid_janelia_static (64ch, 1200s, 72 units, 30KS/s)	fGpu=0	fGpu=1
fParfor=0	Runtime (s): Detect + feature (s): 132.0s Cluster (s): 94.0s Automerge (s): 17.4s Total runtime (s): 243.4s Runtime speed x4.9 realtime memory usage (GiB): detect(GiB): 0.900 sort(GiB): 0.380	Runtime (s): Detect + feature (s): 57.5s Cluster (s): 30.6s Automerge (s): 19.6s Total runtime (s): 107.6s Runtime speed x11.2 realtime memory usage (GiB): detect(GiB): 1.090 sort(GiB): 0.482
fParfor=1 (4 local workers)	Runtime (s): Detect + feature (s): 86.9s Cluster (s): 48.7s Automerge (s): 14.1s Total runtime (s): 149.7s Runtime speed x8.0 realtime memory usage (GiB): detect(GiB): 4.192 sort(GiB): 0.577	CRASHED
fParfor=1 (20 local workers)	Runtime (s): Detect + feature (s): 76.1s Cluster (s): 22.1s Automerge (s): 10.7s Total runtime (s): 108.9s Runtime speed x11.0 realtime memory usage (GiB): detect(GiB): 4.169 sort(GiB): 0.560	CRASHED
fParfor=1 (20 remote workers)	Runtime (s): Detect + feature (s): 58.4s Cluster (s): 19.2s Automerge (s): 9.3s Total runtime (s): 86.9s Runtime speed x13.8 realtime memory usage (GiB): detect(GiB): 4.221 sort(GiB): 0.743	**-p gpu=”gpures:2″** Runtime (s): Detect + feature (s): 38.2s Cluster (s): 12.4s Automerge (s): 9.9s Total runtime (s): 60.5s Runtime speed x19.8 realtime memory usage (GiB): detect(GiB): 4.174 sort(GiB): 0.334

use gaussian kernel smoothing (make sure i get half a fall off at half the mindist). normalize by projecting a uniform field and ensure uniform field back.

advantage of this approach is robustness to where the peak site is located in determining the peak location.

# fixed automerging issue
– Spike indexing was incorrect when extracting trPc 3D array.
– waveform shifting produced comparable result

# todo
– [x] compute rho and delta using parallel resources
– [x] compare performance between irc and irc2
– [ ]add drift correction and compare drift performance

# Runtime comparison
– dataset: static_siprobe\rec_64c_1200s_11
– irc.m: 123s, mean accuracy: 89.8, 62 above .8 accuracy, 1.8GB
– irc2.m (fGpu=1,fParfor=0): 64s, mean accuracy: 89.3, 62 above .8 accuracy, .776GB

# irc2.m speed test (drift correction not implemented yet)
-fGpu=1, fParfor=0: 64s
-fGpu=0, fParfor=0: 389s
-fGpu=0, fParfor=1: 158.6s (20 nodes, local)

# irc2.m test on linux workstation
-fGpu=0, fParfor=0: 292.5s
-fGpu=0, fParfor=1: 133s (20 nodes, remote)
-fGpu=1, fParfor=0: 45s

2019 Sep 22: 93 KG

action: walked for an hour to get to work. Will do the same on the way back. That’s two hours of walking per day. I will also save $20 a day not taking ferry. I will lose 1KG, save $20, and spend extra 1 hour a day commuting to work. and back. In a month I will be 30 KG lighter and $600 richer.