The _id information in a collection entry is really unique?

MarcTabaries · August 19, 2021, 8:39am

Hello,

I need to know if the _id of the entries in a collection are really unique and if there is a risk that the same _id is present in different, or a same, collections.

I plan to use this _id for an e-commerce site project, but I need to be sure that the generated _id is really unique.

I can’t find this information in the documentation.

Thanks in advance.

abernh · August 19, 2021, 8:54pm

This is the function generating your collection documents _ids

github.com

agentejo/cockpit/blob/8856aa2cd2ccf1c92bef6374ba238aa02ca080d3/lib/MongoLite/Database.php#L527

    
      
                              $score  += ($matches / $l);
                          }
                      }
                  }
          
          
    }
          
          
    return $score / \count($needles);
          }
          
          
function createMongoDbLikeId() {
          
          
    // use native MongoDB ObjectId if available
              if (class_exists('MongoDB\\BSON\\ObjectId')) {
                  $objId = new \MongoDB\BSON\ObjectId();
                  return (string)$objId;
              }
          
          
    // based on https://gist.github.com/h4cc/9b716dc05869296c1be6
          
          
    $timestamp = \microtime(true);

It uses if available either

  $objId = new \MongoDB\BSON\ObjectId();

(which is what MongoDB normally uses for their _ids https://docs.mongodb.com/manual/core/index-unique/; see also Generating Globally Unique Identifiers for Use with MongoDB | MongoDB Blog )

or a “mongo db like id”, based on

the timestamp in ms
a “random” 5 char integer
a “random” 2-3 char integer

binaried, hashed and hexed.

Both seem to be setup to create a very unique ID in forever UNLESS you generate

MongoDB: > 16million items within 1 second ( https://betterprogramming.pub/is-the-id-property-in-mongodb-100-unique-f1eaa19522ba#dd87 )
MongoLike: > 88million items at the same microsecond (?*)

*) 89000 * 990 ~ 88mill

MarcTabaries · August 23, 2021, 9:17am

Ok, thanks for this reply !

abernh · August 27, 2021, 12:48pm

There is currently a bug, causing a relative high chance of duplicate IDs for MongoLite on fast import processes.

github.com/agentejo/cockpit

Cockpit generates duplicate _id values

opened 07:47AM - 08 Feb 19 UTC

rootzjg

We have integrated Cockpit to our own system. The system creates collection entr…ies based on what we have in our own database. When starting with an empty Cockpit CMS, the initial import creates thousands of collections entries using the API as fast as possible. This results in duplicate _id values for some (but not all) of the entries that were created at the same second (same timestamp in _created field). ``` SELECT json_extract(Games5c3340b034ba2.document,'$._id') as id, count(*) as count FROM Games5c3340b034ba2 GROUP BY json_extract(Games5c3340b034ba2.document,'$._id') HAVING count > 1; 5c40b42966623700110001b8|2 5c40b4296662370011000389|2 5c40b42966623700120003b1|2 5c40b42a666237001100011c|2 5c40b42c666237001200012f|2 5c40b42c66623700150000fc|2 5c40b42d666237001400009d|2 5c40b42d66623700140001a5|2 5c40b42d6662370016000245|2 5c40b42e6662370014000237|2 5c40b42e666237001500009f|2 5c40b431666237001300005d|2 5c40b438666237001300028b|2 5c40b438666237001500031d|2 5c40b43966623700120001f2|2 5c40b439666237001300001a|2 5c40b439666237001300024e|2 5c40b43c666237001600039a|2 5c40b44066623700140001c7|2 5c40b4446662370011000377|2 5c40b444666237001200030d|2 ``` I checked that these duplicate _id values are found only for entries that have the same creation/modify timestamp. Here's one example (timestamp: 1547744300) where you can see that a couple of entries have the same _id: ``` sqlite> SELECT json_extract(Games5c3340b034ba2.document,'$._id') as id, json_extract(Games5c3340b034ba2.document,'$.gameID') ...> FROM Games5c3340b034ba2 ...> WHERE json_extract(Games5c3340b034ba2.document,'$._created') = 1547744300 ...> order by json_extract(Games5c3340b034ba2.document,'$._id'); 5c40b42c6662370012000019|434 5c40b42c666237001200001e|7710 5c40b42c666237001200005e|763 5c40b42c666237001200007a|607 5c40b42c666237001200007f|1373 5c40b42c66623700120000ab|193 5c40b42c66623700120000bc|787 5c40b42c66623700120000df|1326 5c40b42c666237001200012f|2209 5c40b42c666237001200012f|4028 5c40b42c6662370012000144|1220 5c40b42c666237001200014c|274 5c40b42c666237001200015b|837 5c40b42c666237001200015e|7699 5c40b42c666237001200017c|462 5c40b42c66623700120001a3|4055 5c40b42c66623700120001d9|302 5c40b42c6662370012000220|1490 5c40b42c6662370012000257|131 5c40b42c666237001200027a|6354 5c40b42c666237001200027d|715 5c40b42c666237001200029d|886 5c40b42c66623700120002aa|523 5c40b42c6662370012000309|748 5c40b42c6662370012000321|1111 5c40b42c6662370012000345|1078 5c40b42c666237001200035e|7709 5c40b42c66623700120003a2|2609 5c40b42c66623700120003b9|681 5c40b42c66623700120003d0|2699 5c40b42c6662370013000017|6049 5c40b42c6662370013000043|1492 5c40b42c666237001300004e|2719 5c40b42c6662370013000059|2737 5c40b42c666237001300007f|1231 5c40b42c66623700130000a1|881 5c40b42c6662370013000120|397 5c40b42c6662370013000143|6176 5c40b42c6662370013000157|411 5c40b42c666237001300015f|801 5c40b42c6662370013000185|2542 5c40b42c666237001300019e|6732 5c40b42c66623700130001a4|2600 5c40b42c66623700130001ae|330 5c40b42c66623700130001c2|1292 5c40b42c66623700130001d3|7071 5c40b42c66623700130001e8|959 5c40b42c6662370013000216|2046 5c40b42c6662370013000235|6266 5c40b42c6662370013000286|259 5c40b42c66623700130002b6|1338 5c40b42c66623700130002e9|1165 5c40b42c66623700130002f7|662 5c40b42c6662370013000304|6472 5c40b42c666237001300034e|2030 5c40b42c6662370013000380|1427 5c40b42c6662370013000392|659 5c40b42c6662370013000393|358 5c40b42c6662370013000397|7700 5c40b42c666237001300039b|6831 5c40b42c6662370015000019|366 5c40b42c6662370015000059|2626 5c40b42c6662370015000062|6555 5c40b42c6662370015000080|2717 5c40b42c6662370015000086|1351 5c40b42c66623700150000bb|2604 5c40b42c66623700150000f4|871 5c40b42c66623700150000fc|1155 5c40b42c66623700150000fc|2504 5c40b42c6662370015000104|1235 5c40b42c6662370015000139|1260 5c40b42c6662370015000197|1191 5c40b42c66623700150001fe|937 5c40b42c6662370015000204|694 5c40b42c6662370015000215|6024 5c40b42c666237001500021e|2565 5c40b42c666237001500023c|657 5c40b42c666237001500023d|2650 5c40b42c6662370015000272|910 5c40b42c666237001500028c|872 5c40b42c66623700150002bd|1463 5c40b42c6662370015000320|1508 5c40b42c6662370015000323|420 5c40b42c666237001500032c|7639 5c40b42c666237001500033e|2557 5c40b42c666237001500035a|6063 5c40b42c666237001500038e|196 5c40b42c66623700150003d5|892 5c40b42c66623700150003d8|2017 ``` We are using the latest Docker image.

A PR with a fix was pushed ( fix (Bug 1002:createMongoDbLikeId): microsecond float to big integer … by abernh · Pull Request #1465 · agente ) and should be available soon.

abernh · September 28, 2021, 8:21am

There is currently a bug, causing a relative high chance of duplicate IDs for MongoLite on fast import processes.

Fixed in V 0.12.2, Sept 06, 2021

Topic		Replies	Views
How to enforce unique values in Collections? Support	3	1319	October 26, 2018
Unique id for collection fields to keep track of renamed or deleted fields General	0	869	September 28, 2018
Unique field value in v2 Support	1	264	November 20, 2022
Check if entry is in collection to prevent from duplication entires Support	1	476	October 9, 2020
What do the IDs mean _mby _by _id _o General	1	435	January 28, 2022

The _id information in a collection entry is really unique?

Related topics