ML Community Day is November 9! Join us for updates from TensorFlow, JAX, and more Learn more

tff.simulation.CSVMetricsManager

Utility class for saving/loading experiment metrics via a CSV file.

Inherits From: MetricsManager

csv_filepath A string specifying the file to write and read metrics from.

ValueError If csv_filepath is an empty string.
ValueError If the file at csv_filepath already exists but does not contain a round_num column.

latest_round_num The last round number passed to update_metrics.

If not metrics have been written, this will be None, otherwise it will be a nonnegative integer.

metrics_filename

Methods

clear_metrics

View source

Clear out metrics at and after a given starting round_num.

By using this method, this class can be used upon restart of an experiment at round_num to ensure that no duplicate rows of data exist in the CSV file. This method will atomically update the stored CSV file.

Note that if clear_metrics(round_num=0) is called, all metrics are cleared in a more performant manner. Rather than removing all rows of the CSV with round numbers greater than or equal to round_num, we simply remove all rows of the CSV when round_num=0.

Args
round_num A nonnegative integer indicating the starting round number for clearing metrics from the manager's associated CSV.

Raises
ValueError If round_num is negative.

get_metrics

View source

Retrieve the stored experiment metrics data for all rounds.

Returns
A sequence representing all possible keys for the metrics, and a list containing experiment metrics data for all rounds. Each entry in the list is a dictionary corresponding to a given round. The data has been flattened, with the column names equal to the path in the original nested metric structure. There is a fieldname round_num to indicate the round number.

save_metrics

View source

Updates the stored metrics data with metrics for a specific round.

The specified round_num must be nonnegative, and larger than the latest round number for which metrics exist in the stored metrics data. For example, calling this method with round_num = 3 then round_num = 7 is acceptable, but calling the method with round_num = 6 then round_num = 6 (or anything less than 6) is not supported.

This method will atomically update the stored CSV file. Also, if stored metrics already exist and metrics contains a new, previously unseen metric name, a new column in the dataframe will be added for that metric, and all previous rows will fill in with NaN values for the metric.

The metrics written are the leaf node tensors of the metrics_to_append structure. Purely scalar tensors will be written as scalars in the CSV, while tensors with non-zero rank will be written as a list of lists. For example, the tensor tf.ones([2, 2]) will be written to the CSV as '[[1.0, 1.0], [1.0, 1.0]'.

Args
metrics A nested structure of metrics collected during round_num. The nesting will be flattened for storage in the CSV (with the new keys equal to the paths in the nested structure).
round_num Communication round at which metrics_to_append was collected.

Raises
ValueError If the provided round number is negative.
ValueError If the provided round number is less than or equal to the latest round number in the stored metrics data.