MongoDB 1.6 won’t let you shard a collection without an index on the shard key. I thought the requirement was only for sharding existing collections so after sharding a collection, I dropped the shard key index. MongoDB was perfectly happy letting me do this.
It wasn’t until a couple days later that I noticed my cluster was imbalanced. Investigating the logs, I noticed that all the chunk migrations were failing with the following message:
Tue Nov 30 19:17:33 [migrateThread] things.myCollection Assertion failure ii >= 0 db/dbhelpers.cpp 278
0x534a61 0x54163f 0x6fb852 0x806962 0x7fea55 0x837550 0x7fd6224859ca 0x7fd621a346fd
/mongo/mongod(_ZN5mongo12sayDbContextEPKc+0xb1) [0x534a61]
/mongo/mongod(_ZN5mongo8assertedEPKcS1_j+0x10f) [0x54163f]
/mongo/mongod(_ZN5mongo7Helpers11removeRangeERKSsRKNS_7BSONObjES5_bbPNS0_14RemoveCallbackE+0xc02) [0x6fb852]
/mongo/mongod(_ZN5mongo13MigrateStatus3_goEv+0x6c2) [0x806962]
/mongo/mongod(_ZN5mongo13migrateThreadEv+0xe5) [0x7fea55]
/mongo/mongod(thread_proxy+0x80) [0x837550]
/lib/libpthread.so.0(+0x69ca) [0x7fd6224859ca]
/lib/libc.so.6(clone+0x6d) [0x7fd621a346fd]
Tue Nov 30 19:17:33 [migrateThread] ~ScopedDBConnection: _conn != null
Tue Nov 30 19:17:33 [migrateThread] config change: { _id: "rep2-2-2010-11-30T19:17:33-134", server: "rep2-2", time: new Date(1291144653759), what: "moveChunk.to", ns: "things.myCollection", details: { step1: 7 } }
Tue Nov 30 19:17:33 [migrateThread] ERROR: migrate failed: assertion db/dbhelpers.cpp:278After doing some googling, I discovered I wasn't the only one to receive this error. It was recently discussed on the MongoDB users mailing list.
Turns out, shard keys must be indexed for chunk migration to happen. Without it, chunk migration will fail. Simply creating an index on the shard key fixed the problem and my cluster quickly started passing chunks around to rebalance itself.