Categories: ADMIN GUIDE MAINTENANCE

Backup & Restore

Data Model

step data are organized as collections into MongoDB. Each collections contains a specific type of data, which are explained below :

  • artefacts : the test plans definitions
  • functions : the keyword definitions
  • parameters : the user defined parameters
  • resources : the uploaded files (datasheet, keyword jars etc…)
  • executions : the executions entries, tab Executions > Execution list
  • reports : the detailed executions steps of an execution
  • measurements : any measurements registered during an execution
  • views : additional information about executions (tab Performance, Error etc…)
  • screenInputs : entries defined under the Admin > Settings > Screens
  • users : the registered users into step
  • controllerlogs : some startup logs

Backup

Complete backup

Assuming you are running MongoDB with the default settings (host localhost, port 27017, no authentication), you can perform backups following below instructions.

To perform a full backup of the step database, use the following command to output the backup to the backup folder:

mongodump -d step -o backup

If you prefer create a compressed archive, use the following :

mongodump -d step --gzip --archive=backup.gzip

You should see the following output which confirms the backup succeeded :

mongodump -d step -o backup
2017-10-05T10:45:39.677+0000    writing step.functions to
2017-10-05T10:45:39.677+0000    writing step.reports to
2017-10-05T10:45:39.677+0000    writing step.artefacts to
2017-10-05T10:45:39.677+0000    writing step.views to
2017-10-05T10:45:39.678+0000    done dumping step.functions (6 documents)
2017-10-05T10:45:39.678+0000    writing step.measurements to
2017-10-05T10:45:39.678+0000    done dumping step.reports (3 documents)
2017-10-05T10:45:39.678+0000    writing step.users to
2017-10-05T10:45:39.679+0000    done dumping step.measurements (2 documents)
2017-10-05T10:45:39.679+0000    writing step.controllerlogs to
2017-10-05T10:45:39.679+0000    done dumping step.users (1 document)
2017-10-05T10:45:39.679+0000    writing step.executions to
2017-10-05T10:45:39.680+0000    done dumping step.artefacts (6 documents)
2017-10-05T10:45:39.680+0000    done dumping step.views (3 documents)
2017-10-05T10:45:39.680+0000    done dumping step.controllerlogs (1 document)
2017-10-05T10:45:39.681+0000    done dumping step.executions (1 document)

Follow this link to fully restore your database from such a backup.

Partial backup

To perform a backup of a specific step collection, use the following command :

mongoexport -d step -c <collection_name> -o <collection_name>.json

To display the available collections for backup, connect to you mongo instance and execute the following :

> use step
switched to db step
> db.getCollectionNames()
[
        "controllerlogs",
        "executions",
        "functionPackage",
        "functions",
        "measurements",
        "plans",
        "projects",
        "parameters"
        "reports",
        "resourceRevisions",
        "resources",
        "screenInputs",
        "settings",
        "tasks",
        "users",
        "views"
]
> exit

Follow this link to restore a collection from such a collection export.

Restore

Assuming you are running MongoDB with the default settings (host localhost, port 27017, no authentication) and that an archive previously created using the backup procedure exists, use below instructions to restore your database.

Complete Restore

To fully restore a step database from a backup folder, use the following command :

mongorestore --drop -d step backup

If you prefer to restore the database from a compressed archive previously created, use the following :

mongorestore --drop -d step --gzip --archive=backup.gzip

Warning : using the –drop flag will cause the existing content to be deleted ! You should see the following output which confirms the restore succeeded :

mongorestore --drop backup/
2017-10-05T10:48:29.070+0000    preparing collections to restore from
2017-10-05T10:48:29.089+0000    reading metadata for step.functions from backup/step/functions.metadata.json
2017-10-05T10:48:29.098+0000    reading metadata for step.artefacts from backup/step/artefacts.metadata.json
2017-10-05T10:48:29.148+0000    reading metadata for step.executions from backup/step/executions.metadata.json
2017-10-05T10:48:29.156+0000    reading metadata for step.views from backup/step/views.metadata.json
2017-10-05T10:48:29.265+0000    restoring step.functions from backup/step/functions.bson
2017-10-05T10:48:29.442+0000    restoring step.artefacts from backup/step/artefacts.bson
2017-10-05T10:48:29.584+0000    restoring step.executions from backup/step/executions.bson
2017-10-05T10:48:29.719+0000    restoring step.views from backup/step/views.bson
2017-10-05T10:48:29.720+0000    no indexes to restore
2017-10-05T10:48:29.720+0000    finished restoring step.functions (6 documents)
2017-10-05T10:48:29.720+0000    no indexes to restore
2017-10-05T10:48:29.720+0000    restoring indexes for collection step.executions from metadata
2017-10-05T10:48:29.721+0000    finished restoring step.artefacts (6 documents)
2017-10-05T10:48:29.721+0000    restoring indexes for collection step.views from metadata
2017-10-05T10:48:29.785+0000    reading metadata for step.reports from backup/step/reports.metadata.json
2017-10-05T10:48:30.113+0000    finished restoring step.executions (1 document)
2017-10-05T10:48:30.157+0000    reading metadata for step.measurements from backup/step/measurements.metadata.json
2017-10-05T10:48:30.255+0000    finished restoring step.views (3 documents)
2017-10-05T10:48:30.373+0000    restoring step.reports from backup/step/reports.bson
2017-10-05T10:48:30.382+0000    reading metadata for step.users from backup/step/users.metadata.json
2017-10-05T10:48:30.498+0000    restoring step.measurements from backup/step/measurements.bson
2017-10-05T10:48:30.565+0000    reading metadata for step.controllerlogs from backup/step/controllerlogs.metadata.json
2017-10-05T10:48:30.565+0000    restoring indexes for collection step.reports from metadata
2017-10-05T10:48:30.699+0000    restoring step.users from backup/step/users.bson
2017-10-05T10:48:30.699+0000    restoring indexes for collection step.measurements from metadata
2017-10-05T10:48:30.807+0000    restoring step.controllerlogs from backup/step/controllerlogs.bson
2017-10-05T10:48:31.235+0000    finished restoring step.reports (3 documents)
2017-10-05T10:48:31.316+0000    finished restoring step.measurements (2 documents)
2017-10-05T10:48:31.316+0000    no indexes to restore
2017-10-05T10:48:31.316+0000    finished restoring step.controllerlogs (1 document)
2017-10-05T10:48:31.316+0000    no indexes to restore
2017-10-05T10:48:31.316+0000    finished restoring step.users (1 document)
2017-10-05T10:48:31.316+0000    done

Partial restore

To perform a restore of a specific step collection, use the following command :

mongoimport --drop -d step -c <collection_name> <collection_name>.json

Warning : using the –drop flag will cause the existing collection content to be deleted !

Advanced : backup executions in a defined time frame

As the executions, reports and measurements collections may grow very quickly with the number of tests executions, you may want to export only some of the data associated with executions which ran into a defined time frame.

For this, we need first to export the executions ids present the defined time frame.

As an example, we will export the reports collection using a time frame defined between the 2018/06/06 2 PM and the 2018/06/07 3 PM.

Open a terminal and execute the following command (make sure to replace the start / end times with the values you need) :

mongo localhost/step --quiet --eval "db.executions.find({ '$and': [{'startTime': {'$gt': NumberLong(ISODate('2018-06-06T14:00:00'))}}, {'startTime': {'$lt' : NumberLong(ISODate('2018-06-07T15:00:00'))}}] }).map(e => '\''.concat(e._id.valueOf(),'\'')).toString()" > executions_ids

This will output the execution ids into the file “executions_ids” and its content should look like :

'5b17e8fe064db31178a3c5a1','5b17e936064db31178a3c5f5'

Now that you have the executions ids, you can execute the below commands in a terminal by replacing the executions ids list with the one you just exported :

Export the reports :
mongoexport -d step -c reports -q "{'executionID': {'$in': ['5b17e8fe064db31178a3c5a1','5b17e936064db31178a3c5f5']}}" -o reports.json 

Export the measurements :
mongoexport -d step -c measurements -q "{'eId': {'$in': ['5b17e8fe064db31178a3c5a1','5b17e936064db31178a3c5f5']}}" -o measurements.json

Export the executions :
mongoexport -d step -c executions -q "{'_id': {'$in': [ObjectId('5b17e8fe064db31178a3c5a1'),[ObjectId('5b17e936064db31178a3c5f5')]}}" -o executions.json

See Also