Hello everyone,

I try to run this example in K8s cluster.

https://www.tensorflow.org/federated/tutorials/simulations

I'm using the deploy with 2 pods with PVC by client. The PVC have associate the path to "emnist_all.sqlite". I had use IP from Load Balancer and run the "execute()" function in my local laptop (i.e., server machine).

The "execute()" function return,

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "'329bf95e-e096-4aa0-8ed2-5a76fa53afbd'"
debug_error_string = "{"created":"@1634208969.307600624","description":"Error received from peer ipv4:XX.XXX.XXX.XXX:80","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"'329bf95e-e096-4aa0-8ed2-5a76fa53afbd'","grpc_status":3}"

Function call stack:
pruned
"
debug_error_string = "{"created":"@1634209408.331193271","description":"Error received from peer ipv4:35.241.174.167:80","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Exception calling application: Prepare() failed: [1] no such table: examples: SELECT serialized_example_proto FROM examples WHERE client_id = 'f0000_14' and split_name ='digits_only_train'\n\t [[{{node StatefulPartitionedCall_1/StatefulPartitionedCall/ReduceDataset}}]] [Op:__inference_pruned_16583]\n\nFunction call stack:\npruned\n","grpc_status":2}"

One of the pods returns nothing and the another returns,

Function call stack:
pruned

E1014 11:08:12.058994 139767136110336 _server.py:445] Exception calling application: Prepare() failed: [1] no such table: examples: SELECT serialized_example_proto FROM examples WHERE client_id = 'f0001_41' and split_name ='digits_only_train'
[[{{node StatefulPartitionedCall_1/StatefulPartitionedCall/ReduceDataset}}]] [Op:__inference_pruned_20051]

RuntimeError: The executor service has not yet been configured with cardinalities and cannot execute any concrete requests.
2021-10-14 11:08:22.303941: E tensorflow/core/framework/node_def_util.cc:623] NodeDef mentions attribute metadata which is not in the op definition: Op<name=ParallelMapDatasetV2; signature=input_dataset:variant, other_arguments:, num_parallel_calls:int64 -> handle:variant; attr=f:func; attr=Targuments:list(type),min=0; attr=output_types:list(type),min=1; attr=output_shapes:list(shape),min=1; attr=use_inter_op_parallelism:bool,default=true; attr=deterministic:string,default="default"; attr=preserve_cardinality:bool,default=false> This may be expected if your graph generating binary is newer than this binary. Unknown attributes will be ignored. NodeDef: {{node ParallelMapDatasetV2/_5}}

Can you help me, please? Any idea?

I noticed that the multi-machine version is coming soon...

https://www.tensorflow.org/federated/tutorials/simulations#multi-machine_simulations_on_gcpgke_gpus_tpus_and_beyond

Thanks in advance. Best,