Managing File Storage

By default, SheetKraft stores activity files, input files, and output files in the database. This is convenient from a maintenance perspective, but the database size can grow quite fast if large files are getting stored frequently. Hence, SheetKraft also provides other options for storage of these files.

The value of the FileStorageLocation key in sk.Configuration controls where new files will be stored. The values supported are Database, AmazonS3, or FileSystem. If this key is not present, the value defaults to Database. Note that this only controls new files and not existing files. If this value is changed over time, and the MigrateFileStorage.exe tool is not run, different files will get stored at different places. Such a distributed setup works without any problems, but it is recommended to keep all files at just one place.

If the FileStorageLocation value is set to AmazonS3, the amazonS3Buckets.config is expected to contain a key called SheetKraft with a value containing configuration for an Amazon s3 bucket where files will be stored. This bucket should not be used for any other purpose. Here is an example:

</amazonS3Buckets>

If the FileStorageLocation value is set to AzureBlob, the azureBlobStorage.config is expected to contain a key called SheetKraft with a value containing configuration for an Azure blob storage where files will be stored. This storage should not be used for any other purpose. Here is an example:

</azureBlobStorage>

If the FileStorageLocation value is set to FileSystem, the staticFolders.config file is expected to contain a key called SheetKraft.FileStorage with a value containing a folder path. This folder must be accessible from all application servers, if there is more than one server. Here is an example:

</staticFolders>

It is important to not have invalid configuration with these special keys, as doing so will lead to unexpected errors.

The MigrateFileStorage.exe tool in the Runner folder in the SheetKraft installation helps with moving files from wherever they may be stored (including a combination of locations) into the location configured with the FileStorageLocation key. This tool does not take any arguments. It just moves files from their actual source into the configured target location. It does this one file at a time, and it is safe to just terminate this program at any time. If it is terminated or fails for any reason, it can simply be rerun to resume the moving process.

If any files are moved out of the database by running this tool into locations other than the database, it is essential to keep the configuration for these locations intact and accessible. Otherwise, the system will fail to access files that it needs, leading to various errors. If the migration was done as a temporary arrangement for managing free space, once the space issue is resolved, the FileStorageLocation value should be set to the desired long term location and ManageFileStorage.exe should be run until it completes migrating all files to the long term location. Any intermediate configuration should not be deleted until this is done.