Log in to start a topic

Advice on how to load big files (1 GB and above)

Posted by: achort on 4 April 2023, 1:32 am EST

  • achort

    Posted 4 April 2023, 1:32 am EST

    Hello,

    We are looking at ways to support editing really big files with SpreadJS and I’m reaching out in search of advice.

    1. Given there are several ways to load data I was wondering what would be the best approach to load really big files like for example:

      a) CSV file with 5 million records and 100 columns.

      b) XLSX file with 1 million records and 100 columns.

      (are these sizes reasonable to load in SpreadJS according to your experience?)

    2. Any advice on how to prevent or handle out of memory errors from the browser?.

    3. Does SpreadJS support partially loading a file? In the sense that more data is loaded when user scrolls or moves around the file.

    Thank you.

    Note: These are the ways to load data I’ve found so far:

  • ankit.kumar

    Posted 5 April 2023, 3:22 pm EST

    Hello,

    1. The best ways to load an big excel file is to load it with incremental or lazy mode. In “Incremental” mode, the user can perform UI operation while the file is being loaded in the SpreadJS.

    The file is only loaded when user does not perform any UI operations. On the other hand, when file is loaded with “Lazy” mode, only the minimal and necessary data is loaded at first and other data is loaded in background and when needed. This makes it faster to load big excel files.

    It is important to note that, when you have big excel file, you can store them in SJS format and accordingly load the SJS format file. SJS format is a SpreadJS specific format which is basically a zip folder. It has increased performance for loading big files and it also reduces the size of big files as well.

    1. You can load excel files with lazy mode options in order to prevent the browser from becoming unresponsive and out of memory options errors. Please note that since SpreadJS is a client side based application and its runs on the browser of the client(user), the system configuration of user also play a vital role in the performance of SpreadJS.

    2. Yes, SpreadJS supports loading a file partially. This feature is supported when an excel file is loaded in “Incremental” or “Lazy” mode.

    Additionally, you can utilize spread.import(), spread.export() for importing/exporting excel files and spread.save() and spread.open() methods for importing/exporting SJS format files.

    Please refer to the following docs and demo for more information:

    SJS format: https://www.grapecity.com/spreadjs/docs/features/spreadjs-file-format#import-files

    SpreadJS File Format Demo: https://www.grapecity.com/spreadjs/demos/features/spreadjs-file-format/overview/purejs

    Regards,

    Ankit

  • achort

    Posted 1 May 2023, 8:39 am EST

    Hi, thank you for all the answers.

    I have a few follow up questions regarding the new format SJS.

    1. What options are out there to generate a SJS file? Other than doing spread.import() with a csv/xlsx file and then spread.export() to get the SJS file.
    2. Are there any options to create a SJS file from a CSV/XLSX file in a runtime that is not a browser (ie. NodeJS or any backend runtime)?
    3. Are there any options to create a CSV/XLSX file from a SJS file in a runtime that is not a browser (ie. NodeJS or any other backend runtime)?

    Thank you

  • ankit.kumar

    Posted 2 May 2023, 3:18 pm EST

    Hello,

    Please refer to the below answers to your questions.

    1. Currently, SpreadJS provides spread.save() method to save the spreadsheet in SJS format and spread.open() method of load an SJS format file. It is important to note that spread.import() and spread.export() methods are used to import and export the excel(xlsx) format files respectively.

    2 & 3. We would like to explain you that SpreadJS is a client side application which primarily runs in browser. You can import excel/csv file into SpreadJS and then export them into SJS format. Also, you can import SJS format file into SpreadJS and then export that file to excel/csv format.

    Please note that although you can use SpreadJS in the NodeJS environment by mocking browser environment, it is not officially supported and does not provide all the features of SpreadJS. Refer to the following blog on using SpreadJS with nodejs: https://www.grapecity.com/blogs/how-to-import-export-excel-xlsx-using-node-js

    We would recommend you to use our another product which is Documents for Excel for the backend support. The Documents for Excel has two variants which are Documents for Excel .NET and Documents for Excel Java. These products are compatible with SpreadJS and provide various APIs for different features as they are in SpreadJS.

    Please refer to https://www.grapecity.com/documents-api-excel and https://www.grapecity.com/documents-api-excel-java for more information about Documents for Excel products.

    Regards,

    Ankit

Need extra support?

Upgrade your support plan and get personal unlimited phone support with our customer engagement team

Learn More

Forum Channels

  • © 2023 MESCIUS inc. All Rights Reserved.  ·  1.800.858.2739

    All product and company names herein may be trademarks of their respective owners.