Why would performing an index fast full scan 430 times, reading 38MB and getting back 297 rows cause a query to take almost an hour?
The mongodb.log file for it showed connections coming in, but I simply could not connect to it
In fact I could find no relevant errors in any of the log files…
However this is a rather unintelligible long alphanumeric string…
A quick look at the relevant statistics showed why there was such a significant difference
As you can see, all the waits are on obj# 0, but there is never any object with and object_id of 0
This seemed rather strange as it suggested that the sync source was the most geographically distant replica set member
However, two days later whilst still eating my breakfast I was asked to join a conference call about a bigger problem
As there was a chap from the product looking over my shoulder at the time and I had confidently told him that the problem was…
The situation with the second replica set member was a little more interesting …