(beginner) how does populate work?

olicorn · October 5, 2020, 8:06am

Hello,
thank you for this great work. Unfortunately, I am struggling with the documentation.

I have a collection A which contains collectionLinks to collection B and collection C

is there a way to use populate in order to have only the _id field of the linked collection items instead of their full JSON ? My objective here is to reduce the size of the returned JSON.

Here is the kind of request I make

var finalUrl = baseUrl + complementUrl + tokenParam;
var params = {
    method: 'post',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
        filter: {},
        sort: {lessonNum:1},
        populate: 1, // resolve linked collection items
    })
};

fetch(finalUrl, params)
    .then(res => res.json())
    .then(res => callback(res, args));

thank you very much for your help

raffaelj · October 5, 2020, 11:37am

By default there is no filter option for populated entries, but you can hook into the event system to filter them. You could use the collections.find.after event and unset all data, that you don’t need.

The following snippet for your config/bootstrap.php should perform better, because you set the fields filter before the database request.

<?php

// filter populated items
// I have a collection "software" with a collection link to the collection "repos".
// Change the names to match your setup.

// fire only on api requests and if populate param is true
if (COCKPIT_API_REQUEST && $app->param('populate', false)) {

    // fire only for "software" collection
    $app->on('collections.find.before.software', function($name, &$options) {

        // create event to filter populated "repos" collection
        $this->on('collections.find.before.repos', function($name, &$options) {

            // set the fields filter to prevent populating the whole entry
            // If no custom filter is set, default to "_id" and "full_name",
            // otherwise use the custom filter
            $options['fields'] = $this->param('populateFilter', [
                '_id'       => true,
                'full_name' => true,
            ]);

        });
    });
}

Now you can set custom filters for your populated collection.

var finalUrl = baseUrl + complementUrl + tokenParam;
var params = {
    method: 'post',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
        filter: {},
        sort: {lessonNum:1},
        populate: 1, // resolve linked collection items
        populateFilter: { // populate _id and title
            _id: true,
            title: true
        }
    })
};

fetch(finalUrl, params)
    .then(res => res.json())
    .then(res => callback(res, args));

edit: I added this snippet to my cockpit-scripts repo: https://github.com/raffaelj/cockpit-scripts/blob/master/filter-collection-links/bootstrap.php

olicorn · October 9, 2020, 1:20pm

Thank you very much Raffael,
unfortunately, I can’t make your snippet work.
I tried different things, for example changed :
$app->param(‘populate’, false)) {
to :
$app->param(‘populate’, true)) {

…of course replaced my own collection names in the snippet
also tried to lowcase my collection names.
But the result is still the same : the json still gives me the full data for all the collectionLink

Some hypothesis about the reasons of this :

my “Reference” collection contains two collectionLinks to my “List” collection
I use cockpit Next
maybe my bootstrap.php file is not taken in account ? Is there a way to test this ? (I am really sorry, I am level 0 in php)
cockpit is installed on a subdomain of my website (except for the snippet, everything works perfectly fine)

Again, thank you very much

raffaelj · October 9, 2020, 4:34pm

$app->param('key_name', 'default_fallback') checks for sent parameters and if not present, uses the default fallback. In this case I set the fallback to false for a simple boolean check if the param was sent or not.

This shouldn’t be a problem.

Me too. I always use the next branch.

Create the file path/to/cockpit/config/bootstrap.php with this content:

<?php
echo 'test';

If the string “test” is above your page/above you api response, it works

Can you post your full modified code? Maybe it’s just a simple typo.

olicorn · October 23, 2020, 9:55am

Hello Raffael,
thank you for your answer and sorry for answering so late.

I tried the echo ‘test’ you suggested. when I am sending a request, I get a
“index.html:1 Uncaught (in promise) SyntaxError: Unexpected token e in JSON at position 1”

I think this is because my function expects a JSON to be returned as the result of my request. Am I right ?

Here are some more info about my project :

Here is the ‘fields’ structure of my Reference collection :

fields:
    creator: {name: "creator", type: "collectionlink", localize: false, options: {…}}
    dict_form: {name: "dict_form", type: "text", localize: false, options: Array(0)}
    display: {name: "display", type: "text", localize: false, options: Array(0)}
    exceptionDetails: {name: "exceptionDetails", type: "repeater", localize: false, options: {…}}
    isComplement: {name: "isComplement", type: "boolean", localize: false, options: {…}}
    isConsolidated: {name: "isConsolidated", type: "boolean", localize: false, options: {…}}
    isException: {name: "isException", type: "repeater", localize: false, options: {…}}
    isIAdjective: {name: "isIAdjective", type: "boolean", localize: false, options: {…}}
    isNaAdjective: {name: "isNaAdjective", type: "boolean", localize: false, options: {…}}
    isPublic: {name: "isPublic", type: "boolean", localize: false, options: {…}}
    isStandard: {name: "isStandard", type: "boolean", localize: false, options: {…}}
    isUser: {name: "isUser", type: "boolean", localize: false, options: {…}}
    japanese: {name: "japanese", type: "repeater", localize: false, options: {…}}
    nai_form: {name: "nai_form", type: "text", localize: false, options: Array(0)}
    position: {name: "position", type: "text", localize: false, options: Array(0)}
    premasu_form: {name: "premasu_form", type: "text", localize: false, options: Array(0)}
    publicLists: **<- these are the collectionLink I'd like to modify**
        localize: false
        name: "publicLists"
        options: {link: "List", display: "name", multiple: true, limit: false}
        type: "collectionlink"
    romaji: {name: "romaji", type: "repeater", localize: false, options: {…}}
    ruby: {name: "ruby", type: "repeater", localize: false, options: {…}}
    standardLists: **<- these are the collectionLink I'd like to modify**
        localize: false
        name: "standardLists"
        options: {link: "List", display: "name", multiple: true, limit: false}
        type: "collectionlink"
    ta_form: {name: "ta_form", type: "text", localize: false, options: Array(0)}
    te_form: {name: "te_form", type: "text", localize: false, options: Array(0)}
    translation: {name: "translation", type: "repeater", localize: true, options: {…}}

Here is the code I use to send my request :

class Request
{
    constructor(table, callback, args)
    {
        this._table = table;
        this._callback = callback;
        this._args = args;
        this._typeUrlArg = "/collections/entries/";
        this._url = Request.baseUrl + this._typeUrlArg + Request.token;
        this._params = {
            method: 'post',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({
                filter: {},
                populate: 1, // resolve linked collection items
            })
        };
        Request.token = "?token=XXX";
        Request.baseUrl = 'https://XXX/cockpit/api';
    }
    
    
    /**
    *   executes the request
    */
    execute()
    {
        console.log('request', this._params)
        fetch(this.url, this._params)
            .then(res => res.json())
            .then(res => RequestManager.ResolveCallback(res, this._args));
    }
}

class RequestLoadCollection extends Request
{
    constructor(table, callback, args)
    {
        super(table, callback, args);
        this._type = "LoadRequest";
        this._typeUrlArg = "/collections/entries/";
        this._url = Request.baseUrl+this._typeUrlArg+this._table+Request.token;
        this._params.body = {filter:{},
                             populate:1,
                             populateFilter:{_id:true}}
    }   
}

Here is my bootstrap.php code :

// fire only on api requests and if populate param is true
if (COCKPIT_API_REQUEST && $app->param('populate', false)) {

    // fire only for "Reference" collection
    $app->on('collections.find.before.Reference', function($name, &$options) {
        
        // create event to filter populated "repos" collection
        $this->on('collections.find.before.List', function($name, &$options) {

            // set the fields filter to prevent populating the whole entry
            // If no custom filter is set, default to "_id" and "full_name",
            // otherwise use the custom filter
            $options['fields'] = $this->param('populateFilter', [
                '_id'       => true,
                'name'       => true
            ]);

        });
    });
}

Here what my request looks like when I log it in the console :

{method: "post", headers: {…}, body: {…}}
    body: 
        filter: {}
        populate: 1
        populateFilter: {_id: true}
        __proto__: Object
    headers: 
        Content-Type: "application/json"
        __proto__: Object
    method: "post"
        __proto__: Object

And finally, here is an extract of the result of my request

entries: Array(888)
    [0 … 99]
    0:
        creator: {name: "admin", password: "XXX", email: "XXX", references: "", lists: "", …}
        dict_form: ""
        display: "先（せん）生（せい） professeur, instituteur, maître"
        exceptionDetails: ""
        isComplement: false
        isConsolidated: false
        isException: false
        isIAdjective: false
        isNaAdjective: false
        isPublic: true
        isStandard: true
        isUser: false
        japanese: [{…}]
        nai_form: ""
        position: "A007"
        premasu_form: ""
        publicLists: []
        romaji: [Array(0)]
        ruby: [{…}]
        standardLists: Array(1)
            0:
            creator: {name: "admin", password: "XXX", email: "XXX", references: "", lists: "", …}
            description: ""
            description_en: null
            description_pt: null
            isComplement: false
            isPublic: true
            isStandard: true
            isUser: false
            lessonNum: "01"
            name: "Leçon 1"
            name_en: "Lesson 1"
            name_pt: "Lição 1"
            references: ""
            _by: "5f6c617265363265d20000b0"
            _created: 1600942466
            _id: "5f6c718234616686190000d5"
            _mby: "5f6c617265363265d20000b0"
            _modified: 1600942967
            __proto__: Object
            length: 1
            __proto__: Array(0)
            ta_form: ""
            te_form: ""
            translation: [{…}]
            type: "Nom"
            _by: null
            _created: 1600965146
            _id: "5f6cca1a62393249e7000351"
            _modified

Thank you again for your help, I really appreciate

raffaelj · October 23, 2020, 11:08am

Yes.

I’m sorry, but your console output is hard to read. Could you create a smaller test case? If it works with test collections, that only have 3 fields, you can adopt it to the large collections later.

My code worked in a local cockpit installation and I tested the api requests with Postman. This way you don’t have to care about javascript errors while fiddling with the correct data response.

olicorn · October 23, 2020, 3:42pm

Thank you, I didn’t know Postman.

I created 2 test collections, I can see in Postman that whether I put {populate:1} in my request or not, I get different results.

With {populate:1} :

    "entries": [
        {
            "name": "list verbs",
            "secondary": [
                {
                    "name": "parler",
                    "japanese": "hanasu",
                    "french": "parler",
                    "portuguese": "falar",
                    "_mby": "5f6c617265363265d20000b0",
                    "_by": "5f6c617265363265d20000b0",
                    "_modified": 1603458106,
                    "_created": 1603458106,
                    "_id": "5f92d43a3564398a640002bb",
                    "_link": "TestSecondary"
                },
                {
                    "name": "manger",
                    "japanese": "taberu",
                    "french": "manger",
                    "portuguese": "comar",
                    "_mby": "5f6c617265363265d20000b0",
                    "_by": "5f6c617265363265d20000b0",
                    "_modified": 1603458115,
                    "_created": 1603458076,
                    "_id": "5f92d41c63306286f90001f4",
                    "_link": "TestSecondary"
                }
            ],
            "_mby": "5f6c617265363265d20000b0",
            "_by": "5f6c617265363265d20000b0",
            "_modified": 1603458508,
            "_created": 1603458161,
            "_id": "5f92d471623532e6da000321"
        }
    ],

without {populate:1}

"entries": [
        {
            "name": "list verbs",
            "secondary": [
                {
                    "_id": "5f92d43a3564398a640002bb",
                    "link": "TestSecondary",
                    "display": "parler"
                },
                {
                    "_id": "5f92d41c63306286f90001f4",
                    "link": "TestSecondary",
                    "display": "manger"
                }
            ],
            "_mby": "5f6c617265363265d20000b0",
            "_by": "5f6c617265363265d20000b0",
            "_modified": 1603458508,
            "_created": 1603458161,
            "_id": "5f92d471623532e6da000321"
        }
    ],

I can’t get this behavior on my original Collections, omitting {populate:1} doesn’t change anything, I still get the full data.
Any clue ?

raffaelj · October 23, 2020, 8:11pm

That’s the desired behaviour.

Are your changes in config/bootstrap.php still active? If yes, that might be the reason. Delete the code and start over with the test collection to filter the populated data.

olicorn · October 24, 2020, 10:08am

yes, of course.
Even when I omit {populate:1} all my collectionsLink are still populated, including the collectionLinks in my collectionLinks (2 levels).

I a really puzzled by this difference of behaviour between my actual collections and the test collections…

olicorn · October 24, 2020, 10:50am

ok, found why !
The problem comes from the way I filled my database.
I used javascript to iterate through an old database to store all my Collections in Cockpit.

When I link manually a CollectionLink through the Cockpit UI, {populate} behaviour is ok.
But with the data I stored through javascript, everything works except for populate behaviour.
(I mean I actually have the same display in the cockpit UI for my javascript-created collectionLinks as for my UI-created ones including collectionLink fields)

I think the way I store the collectionLinks is wrong, but I can’t figure how to achieve this.

When I send a fetch request to path/to/cockpit/api/collections/save/CollectionName, here is what my request’s body looks like :

data = {
  field1: 'text',
  collectionLinkField1: {
    _id:'XXX',
    _link:'collection2Name'
  }
}

can you see what’s wrong ?
(sorry for the bother, think I will soon see the light)

raffaelj · October 24, 2020, 12:14pm

I expect, that the _id is correct.

Try it without the underscore in the link key

data = {
  field1: 'text',
  collectionLinkField1: {
    _id:'XXX',
    link:'collection2Name'
  }
}

or also as array

data = {
  field1: 'text',
  collectionLinkField1: [{
    _id:'XXX',
    link:'collection2Name'
  }]
}

The usage of the keys _link and link is one reason, why I actually don’t really like the collectionlink fields. If your linked collection has a field with the name link, the data could mess up.

olicorn · October 24, 2020, 12:20pm

Great ! it is working now !
Thank you very much for your help Raffael, it is very precious

Topic		Replies	Views
Filtering on a collectionlink Support	14	7166	September 8, 2021
Using the API to add new links using only _id Support	1	351	February 15, 2022
Quick Search/Filter Content link Columns in the CMS Support	13	366	December 27, 2024
Calling populate method in own endpoint Support	1	659	July 29, 2019
Collection-Preview: Populate collectionlinks Feature Request	0	843	December 4, 2018

(beginner) how does populate work?

Related topics