irc2 post merging using position and amplitude of clusters

use gaussian kernel smoothing (make sure i get half a fall off at half the mindist). normalize by projecting a uniform field and ensure uniform field back.

advantage of this approach is robustness to where the peak site is located in determining the peak location.

gaussian kernel convolved, maximum slope at the minimum separation distance (sigma=d_min)
inferring spike position using PC1 is more precise than using other components
Great study music helping me to focus

irc2 development log

# fixed automerging issue
– Spike indexing was incorrect when extracting trPc 3D array.
– waveform shifting produced comparable result

# todo
– [x] compute rho and delta using parallel resources
– [x] compare performance between irc and irc2
– [ ]add drift correction and compare drift performance

# Runtime comparison
– dataset: static_siprobe\rec_64c_1200s_11
– irc.m: 123s, mean accuracy: 89.8, 62 above .8 accuracy, 1.8GB
– irc2.m (fGpu=1,fParfor=0): 64s, mean accuracy: 89.3, 62 above .8 accuracy, .776GB

# irc2.m speed test (drift correction not implemented yet)
-fGpu=1, fParfor=0: 64s
-fGpu=0, fParfor=0: 389s
-fGpu=0, fParfor=1: 158.6s (20 nodes, local)

# irc2.m test on linux workstation
-fGpu=0, fParfor=0: 292.5s
-fGpu=0, fParfor=1: 133s (20 nodes, remote)
-fGpu=1, fParfor=0: 45s

Losing weight by walking to work and back

2019 Sep 22: 93 KG

action: walked for an hour to get to work. Will do the same on the way back. That’s two hours of walking per day. I will also save $20 a day not taking ferry. I will lose 1KG, save $20, and spend extra 1 hour a day commuting to work. and back. In a month I will be 30 KG lighter and $600 richer.

sep 19 2019 @ flatiron

Start: 10:30 AM
Goals: memory loop plot, data backup, v4.9.5 debug with bapun,

# Memory test status
Still going. the param_set2.prm include cached results so I need to consider file read time, which should be about 1GB/s.

# Bapun v4.9.5 vs v4.9.11 comparison
No obvious difference. Formatting issue is suspected. Run his dataset tomorrow using his own parameter. Also run using makeprm command

# Dan English Dataset library
Not downloaded yet. I should do this after making parforeval command

# Disk backup
New 4-bay disk is setup. Each can hold 40TB (RAID0). My personal data will be RAID5 (30TB) and the recordings will be RAID0 and will be stored in CEPH. Linux gets 20TB of scratch (RAID0) to be managed by LVM. Eventually I will invest in 14TBx4 which gives 12TB extra with RAID5. This will come at $1600 price tag. I can only afford an enclosure (30TB) at this point. I will setup 10TB scrach drive in linux and 10TB in windows. I will keep a copy of the data at home to be used with my Lenovo.

## Final goal
– 40TB Hitachi RAID5 keep at home (enclosure ordered, fill with personal data)
– 40TB WD RAID0 keep @ work (filled with recordings, hardware RAID)
– 40TB WD RAID0 windows enclosure (temp data keeping purpose)
– 20TB Hitach keep in Moneybox (10TB for windows, 10TB for ubuntu)

## Data migration plan
– day0: empty 80TB WD RAID5 tower to 40TB RAID0 Hitachi (recordings) and 20TB Hitachi drives (personal)
– day1: copy RAID0 Hitachi (recordings) to CEPH via Globus (over the weekend, ~30TB)
– day2: bring 4-bay enclosure from home (Monday), add 40TB WD, copy from 40TB Hitachi RAID0 overnight
– day2: Change 40TB Hitachi to RAID5 and copy personal data (20TB Hitachi) overnight
– day3: Take 40TB Hitach RAID5 home, borrow 40TB WD to setup linux RAID

Flatiron workstation LVM RAID setup

# Dell precision linux setup (T7910)
– Dylan suggested RAID5 XFS on the CENTOS workstation. RAID is managed by LVM (logical volume manager).
– The plan is generate the dataset and run the benchmark on this workstation. When my own linux box is cleared then I could clone the setup.
– Dylan disabled the last access time field, a performance hack.
– Currently generating the test dataset. The RAID is being built but the writing to disk is still possible. He recommended softraid.
– The question about why not using CEPH. read/write variability on the workstation. this is the reason why I want a local RAID. I am dealing with a large file read and write and I want to keep all the processing to local.

# Linux thunderbolt
install sudo apt install bolt
restart and mount
Dell workstation has Thunderbolt port available.

# Memory profiling caveat
In Ubuntu, the memory profiling precision above 20MB (MB is 10^6 whereas MiB is 2^20)

parenting and discipline

Pure freedom is like random noise which knows no direction, no outcome no future. It only leads to a random walk. I am getting concerned about whom my son is becoming. He is getting out of control by yelling, complaining, whining and crying to get his way. The main issue is that I need to get to work on time but my commute often gets delayed by listening to all his demands or my wife’s demands. This is very bad for the future of our family. I have no interest in raising a son whom I would hate to spend time with. Discipline must be balanced with indulgence and many Americans confuse indulgence with happiness. Indulgence is mostly based on material gains or service received by spending money. We need to indulge at times but it must come as reward for work being accomplished, not by accumulating credit card debts. Jordan Petersen said something along the line of “women indulges and man disciplines. Women do as they please and men do what he must”.

I don’t want my son to grow up like a girl. Americans used to be very disciplined people and their hard work changed all of our lives for good. since the civil rights movement the pendulum swung the opposite direction. entitlement and identity politics. disciplines do not matter any more only loud mouth gets what they want. no wonder the country is falling apart. there is no social familial responsibilities. everyone does as they please without thinking of consequences in their near term and far future. the collapse is guaranteed.

I used to be liberal but i can see how the extreme liberalism can destroy individuals families society nation and the whole planet. freedom is a random noise from neuroscience research. it is unpredictable and animal behaviors have been shaped, not perfectly nor completely by the forces of evolution. behavioral flexibility and freedom comes at a cost of suboptimality. micromanaging one’s action is not what i want. i want guidance in everyone’s lives to bring collectively better future. many have tried or at least gave a lip service to this notion. national socialism, international communism, market democracy. market democracy is about choices. females are about choices. shoppers choose. consumers get manipulated by ads and the whole population can be hacked in their brain. what kind of future do i want to live? what do i want my next year to be? am i stuck or am i going somewhere?

don’t be hacked by the social machineries of mass media. news tells half truth half lies. the wise one knows this. the cunning one knows this too. knowledge is power. actionable knowledge is power if i must add. i learned the basic human nature when i restrained by social constructs. it leads to good things in some and terrible outcomes for most people. that’s what an ultimate freedom brings.

I already wasted countless time getting hacked. i am wiser and i will act on the personal experiences i have gained and countless failures i experienced. i am sharing my thoughts to find friends and guide people whom seek advices.

Parenting completes a msn’s destiny. parenting forces me to shape up myself if i managed to be a good dad. what is being a good dad about? i must provide mental and material and emotional supports to my son. i must teach him what he can and cannot eat or play with or touch or act on. i must teach him patience and appreciation for the family and community. i must teach him true strength and warmth of character. i lack many of these but i want my son to be better than me. by helping him to achieve these goals i will become a better man. super man. ubermensh. quiet frankly the future is going to be harder much harder to live in. it would require superhuman qualities to simply survive. i must guide my son to the path toward superhuman. being good is not good enough anymore. being better is not good enough. superhuman intelligence emotional resilience and strength and endurance all of them are required in one person. human being are finally being forced to become complete. we must master our technology not the other way around. we must console machines not the other way around. there are already very few masters and many ignorant mass who live in a flat world. masters live in a tall and narrow spire whom oversees everyone. flatlanders are not even aware of what’s above us. they spend majority of time moving horizontally not vertically.

i need to move vertically and it’s going to require concentrated and directed energy that’s sharply focused toward above.

memory profiling for ironclust

# Goal: expand # channels and durations by tiling and see how the memory use scales per channel and per duration basi

[x] ## 1. expand datasets by tiling.
– cache the original (64ch, 1200s), and tile it

[x] ## 2. run the memory benchmark, parameter loop
– parameter loops: {param_set0.prm, param_set1.prm}
try to break the memory limit (256GB in my workstation)
~1MB/chan/min (3.43 MB/chan/min raw, 153GB/380ch/2hrs)

[x] ## 3. plot the scalability benchmark
– memory, speed, accuracy as a function of channels and duration

# Data storage problem on thunderbolt
[x] order thunderbolt enclosure
[ ] migrate
[ ] 18.04 has thunderbolt driver already mounted
do not install sudo apt install bolt
[ ] confirm interoperability between desktop and notebook

In Ubuntu, the memory profiling precision above 20MB (MB is 10^6 whereas MiB is 2^20)
Dell precision (T7919) could have a thunderbolt 2 card installed.
(compare 1.3 GB/s vs 50 MB/s over the network)

# IronClust remaining work
1. documentation
2. benchmark scalability, memory, accuracy, channel count, density, recording duration
3. make it faster using remote slurm interface

# Ironclust result section
## accuracy
– accuracy comparison
– probe layout comparison. density. columns
– drift handling comparison

## scalabiilty
– scalability plot: memory and runtime scaling
– accuracy vs. speed plot
– speed comparison with other software
– speed scalability using multiple computing cores

## usability
-input and output format