JavaScript MongoDB Querying Large Collections mongodb query large data,mongodb query larger than,mongodb large

I have a MongoDB collection called 'Prices' that I am trying to query for prices between a 'startDate' and 'endDate'.

The collection stores prices every 10 seconds, however when querying this collection for graphing purposes only the price every 1-2 minutes is really important.

I've tried to write this query in two different ways:

Method 1: Using {$gte: startDate, $lte: endDate}

 function getResultsInRange(startDate, endDate) {
    if(typeof startDate !== 'undefined' && typeof endDate !== 'undefined') {
        Price.find({timestamp: {$gte: startDate, $lte: endDate}}, 'price timestamp exchange')
        .sort('-timestamp')
        // .populate('exchange')
        .exec(function(err, prices) {
            if(err) {
                res.jsonp({'error': err});
            } else {
                console.log("Found: " + prices.length + " prices");
                res.jsonp(prices);
            }
        });

    }
}

This method throws the following error:

{"error":{"name":"MongoError","$err":"Executor error: Overflow sort stage buffered data usage of 33554490 bytes exceeds internal limit of 33554432 bytes","code":17144}}

If I remove the sort('-timestamp") line, and run this query again I get the following error:

GET /prices/graph/minute - - ms - -
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

I have also tried using indexes in this query to try and avoid the 32MB in memory sorting limitation, but haven't been able to get it going. I am wondering if it's caused by the issue:

"The total size of an index entry, which can include structural overhead depending on the BSON type, must be less than 1024 bytes."

described here: (http://docs.mongodb.org/manual/reference/limits/#indexes)

Method 2: Using a While Loop to Query for thte Last Price Each X Minutes (eg. 2)

function getResultsInRange(startDate, endDate) {
    if(typeof startDate !== 'undefined' && typeof endDate !== 'undefined') {

        var currentDate = startDate;
        currentDate.setSeconds(0);

        var priceData = {};

        while(currentDate < endDate) {                
            Price.findOne({'timestamp': {$lte: currentDate}}, 'price timestamp exchange')
            .sort('-timestamp')
            .exec(function(err, price) { 
                if(err) {
                    console.log('ERROR: ' + err);
                } else if (price !== null) {               
                    // PUSH DATA TO ARRAY HERE                        
                }
            });

            // Increment Current Date.
            currentDate.setMinutes(currentDate.getMinutes() + 2);
        }

        res.jsonp(priceData);
    }//end if startDate and endDate are defined (giving us a valid date range).
}// end getResultsInRange()

However, this method does not work, it seems like for each iteration of the loop 'currentDate' = 'startDate' so it's only querying for the last price recorded in the database before 'startdate'.

Edit: Method 3: Using Stream() I've also tried using .stream().

 var query = Price.find({timestamp: {$gte: startDate, $lte: endDate}}, 'price timestamp exchange').populate('exchange').stream();
 query.on('data', function(price) {
     // ADD TO ARRAY
 }).on('error', function(err) {
     console.log("ERROR: " + err);
 }).on('close', function() {
     res.jsonp(priceData); 
 });

Any help would be much appreciated!

Answer:1

I'm doing something like: $.ajax({ type: 'POST', dataType: 'json', contentType: 'application/json', url: '/Modal/getModalInfo', data: '{"machine":"' + machine + '"}', success:...

I'm doing something like: $.ajax({ type: 'POST', dataType: 'json', contentType: 'application/json', url: '/Modal/getModalInfo', data: '{"machine":"' + machine + '"}', success:...

  1. ajax call returning object object
  2. ajax call returning empty response
  3. ajax call returning html page
  4. ajax call returning html instead of json
  5. ajax call returning error
  6. ajax call returning undefined
  7. ajax call returning 302
  8. ajax call returning 404
  9. ajax call returning html
  10. ajax call returning json data
  11. ajax call returning 500 error
  12. ajax request returning full page code
  13. ajax request returning undefined
  14. ajax request returning 404
  15. ajax request returning object
  16. ajax call not returning success or error
  17. ajax call not returning response
  18. ajax call not returning data
  19. ajax call not returning to success
  20. ajax call not returning view

I want to override console.log method to invoke a set of tasks whenever console.log is called. I referred other Stackoverflow answers but that give me the error: Uncaught RangeError: Maximum call ...

I want to override console.log method to invoke a set of tasks whenever console.log is called. I referred other Stackoverflow answers but that give me the error: Uncaught RangeError: Maximum call ...

  1. override javascript console.log
  2. override console js
  3. javascript override console.error
  4. javascript override console.log function
  5. javascript override console

I'm writing a form in Angular and I have one criteria that requires that I need to check for that requires the input to ONLY contain letters and numbers AND the first character cannot be a number. Is ...

I'm writing a form in Angular and I have one criteria that requires that I need to check for that requires the input to ONLY contain letters and numbers AND the first character cannot be a number. Is ...

  1. angular form control validation
  2. angular form control set value
  3. angular form control disabled
  4. angular form control example
  5. angular form control get value
  6. angular form control reset
  7. angular form control name
  8. angular form control checkbox
  9. angular form control default value
  10. angular form control valuechanges
  11. angular form control array
  12. angular form control custom validator
  13. angular form control radio button
  14. angular form control seterrors
  15. angular form control clear value
  16. angular form control readonly
  17. angular form control focus
  18. angular form control change event
  19. angular form control label
  20. angular form control type

I have this Ajax to send multiple images: $('#btn').on("click", function () { var formData = new FormData($("#form1")[0]); var path = "php/upload/adm_prodpictures.php"; $....

I have this Ajax to send multiple images: $('#btn').on("click", function () { var formData = new FormData($("#form1")[0]); var path = "php/upload/adm_prodpictures.php"; $....

  1. javascript variable form data
  2. ajax send formdata and variable