Speeding up search on whatanime.ga !
Added 2018-04-02 15:33:48 +0000 UTCNew testing site on https://beta.whatanime.ga
I've been testing a new beta site with improved performance on search speed.
A copy of the 130GB solr core from http://whatanime.ga is split into 10 smaller cores (10-16GB each, with 59-89 million hashes). Each search queries all 10 solr cores in parallel and then merge back. Early tests shows that it can query 950 million image hashes in 1.35 seconds! Almost 10 times as much than the existing one.
I've rewritten a lot of backend scripts into a nice CLI tool. I'm going to publish it on github later this month, after I've completed the docs and tutorial. For experienced developers, it will be a lot easier to setup their own video reverse search engine.
